Efficient invariant feature subspace recovery for domain generalization

Domain generalization aims to learn a machine learning model over multiple training environments to generalize well to unseen test environments. With the advent of large pretrained models, recent work focuses on efficient domain generalization to maximize performance under minimal data and model fine-tuning requirements. Recently, Wang et al. proposed Invariant Feature Subspace Recovery (ISR-Mean), an efficient domain generalization algorithm which uses means of the class-conditional data distributions to provably identify the domain-invariant feature subspace under a given causal model. However, ISR-Mean is not applicable in the realistic setting of multi-class classification as it only utilizes information from a single class, failing to account for information across multiple classes. Further, it is unclear how this notion of invariant subspace recovery can be applied to regression, when there are no class labels. Motivated by the need to enable efficient robustness via invariant feature subspace recovery in these scenarios, this work extends ISR to two novel settings of multi-class classification and regression. First, in multi-class classification, a more general causal model is proposed under which the ISR-Multiclass algorithm is introduced: ISR-Multiclass can provably recover the invariant feature subspace in \lceil d_spu/k \rceil + 1 environments where d_spu is the dimensionality of spurious features and $k$ is the number of classes. Thus, ISR-Multiclass leverages class information to \textit{improve} the environment complexity by a factor of $k$ as compared to the original ISR-Mean, which requires d_spu + 1 environments. Next, in the setting of regression, ISR-Regression is introduced as a provable recovery algorithm under a new causal model for regression. ISR-Regression can identify the invariant feature subspace in d_spu + 1 environments, matching that of the original ISR algorithm. Empirically, ISR-Multiclass and ISR-Regression demonstrate superior performance across new synthetic linear benchmarks (in line with the theoretically claimed environment complexity) and significantly improve the robustness of neural networks trained with various methods (such as ERM, IRM) across synthetic and real-life datasets encoding strong spurious correlations - thus uncovering the generality, accuracy and efficiency of this framework.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Balasubramaniam, Gargi

Contributors dc:contributor

• Zhao, Han

• Domain Generalization
• Out-of-domain (OOD) Generalization
• Invariant Feature Learning
• Spurious Correlations

Statement dc:rights

• Copyright 2023 Gargi Balasubramaniam

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/120084

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F120084