Constrained estimation problems are prevalent in statistics, machine learning, and engineering. These problems encompass constrained generalized linear models, constrained deep neural networks, physics-inspired machine learning, algorithmic fairness, and optimal control. However, existing estimation methods under hard constraints rely on either projection or regularization, which may theoretically exhibit optimal efficiency but are impractical or unreasonably fail in reality. This talk aims to bridge the significant gap between practice and theory for constrained estimation problems.

I will begin by introducing the critical methodology used to bridge the gap, called Stochastic Sequential Quadratic Programming. We will see that SQP methods serve as the workhorse for modern scientific machine learning problems and can resolve the failure modes of prevalent regularization-based methods. I will demonstrate how to make SQP adaptive and scalable using various modern techniques, such as stochastic line search, trust region, and dimensionality reduction. Additionally, I will show how to further enhance SQP to handle inequality constraints online.

Following the methodology, I will present some selective theories, emphasizing the consistency and efficiency of the SQP methods. Specifically, I will show that online SQP iterates asymptotically exhibit normal behavior with a mean of zero and optimal covariance in the Hajek and Le Cam sense. Significantly, the covariance does not deteriorate even when we apply modern techniques driven by practical concerns. The talk concludes with experiments on both synthetic and real datasets.