TOWARDS EFFICIENT LARGE-SCALE BI-LEVEL OPTIMIZATION, ALGORITHMS AND APPLICATIONS

Bi-level optimization is a mathematical framework with a long history of research, dealing with hierarchical optimization problems where one problem is nested within the other. Recently, with the rise of machine learning, bi-level optimization has regained attention as a theoretical framework covering a wide range of machine learning problems, including hyperparameter optimization, neural architecture search, robust machine learning, meta-learning, and physics-informed machine learning. In recent years, as the scale of machine learning problems has grown rapidly, large-scale bi-level optimization has emerged as a critical area of study. These large-scale settings bring unique computational and algorithmic challenges, including memory constraints, scalability issues, and the complexity of high-order gradient computations. This thesis focuses on analyzing the key challenges introduced by large-scale bi-level optimization and presents recent methodological advances aimed at addressing them. Beyond a survey of existing approaches, we propose principled algorithmic designs tailored to various problem scenarios, enabling the efficient resolution of large-scale bi-level optimization tasks in practical applications. These contributions aim to bridge the gap between theoretical developments and scalable deployment in real-world machine learning systems.

Author dc:creator

• SHEN QIANLI

• machine learning
• large-scale
• bi-level Optimization

Licence

• https://scholarbank.nus.edu.sg/bitstreams/552b8cef-27ff-4174-9661-6df57fdc733c/download