Reliable and efficient machine learning under distribution shifts

Over the past decade, machine learning has emerged as a transformative technology, driving advancements across various domains including computer vision, natural language processing, biology, autonomous driving, and recommendation systems. These innovations have been fueled by the development of increasingly sophisticated algorithms, the availability of large-scale datasets, and the growth of computational power. However, a crucial challenge persists in applying machine learning models to real-world scenarios: their performance tends to degrade significantly when confronted with distribution shifts. This dissertation primarily focuses on two types of distribution shifts: task distribution shifts and data distribution shifts. In addressing task distribution shifts, we first develop a theoretical framework to analyze gradient-based meta-learning algorithms from an optimization perspective, leveraging the Neural Tangent Kernel (NTK) as a theoretical tool. We then bridge multi-task learning (MTL) and meta-learning within this NTK-based framework, providing a unified understanding of these learning paradigms. Finally, we tackle a real-world challenge of aligning large language models with diverse user preferences by developing a novel MTL algorithm called Directional Preference Alignment. Regarding data distribution shifts, we first investigate the paradigm of gradual domain adaptation (GDA) through theoretical analysis. This leads to a deeper understanding of GDA and inspires a novel algorithm: Generative Gradual Domain Adaptation with Optimal Transport. Further, we consider the domain generalization setup and develop a new class of provable algorithms, Invariant-feature Subspace Recovery, which can be used to mitigate spurious correlations learned by machine learning models. Finally, we introduce a previously unexplored challenge of data distribution shifts, which we term compositional generalization in the context of multi-domain multi-class classification. To address this challenge, we design a theory-principled fine-tuning algorithm, Compositional Feature Alignment, aimed at enhancing the compositional generalization ability of pre-trained models. Through these contributions, this dissertation advances the field of machine learning by improving model reliability and efficiency under task and data distribution shifts, and bridges the gap between theoretical frameworks and practical applications in various real-world application scenarios.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Wang, Haoxiang

Contributors dc:contributor

• Zhao, Han
• Li, Bo
• Schwing, Alexander Gerhard
• Raginsky, Maxim

• Machine Learning
• Deep Learning
• Distribution Shift

Statement dc:rights

• Copyright 2024 Haoxiang Wang

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/127492

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F127492