Research map / Label-efficient and robust learning / Test-time adaptation
Sufficient Invariant Learning for Distribution Shift
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sufficient Invariant Learning (SIL) and ASGDRO, a sharpness-aware group DRO that seeks common flat minima across environments so models rely on diverse invariant features, evaluated on a new H-CMNIST benchmark plus subpopulation, wilds, and DomainBed tasks.
Problem
Invariant learning methods often learn only a subset of invariant features, so models degrade under distribution shift when some invariant features are unobserved at test time.
Approach
Define Sufficient Invariant Learning, which requires robustness across subsets of invariant features, and approximate it by ASGDRO: a group DRO objective with adaptive sharpness-aware perturbations that drives convergence to flat minima common to all environments, with a normalization operator for scale invariance.
Key results
- On DomainBed, ASGDRO reaches 65.9 average accuracy, the highest among compared methods, and applied to DPLCLIP it raises average to 74.0 from 72.6.
- On H-CMNIST TestBed 2 (Spu & Shape), ASGDRO scores 66.62% versus 57.53% for GDRO and 34.64% for ERM.
- On subpopulation shift, ASGDRO achieves the best worst-group accuracy on CMNIST (74.2%), Waterbirds (91.4%), and CelebA (91.0%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ASGDRO instead of GDRO | H-CMNIST · TestBed 2: Shape | accuracy: 61.44 ± 1.03 -> 69.17 ± 6.19 | Helped |
| Use ASGDRO instead of GDRO | subpopulation shift models · Waterbirds | worst group accuracy: 90.6% -> 91.4% | Helped |
| Use ASGDRO instead of GDRO | subpopulation shift models · CelebA | worst group accuracy: 87.2% -> 91.0% | Helped |
| Use ASGDRO instead of GDRO | subpopulation shift models · CivilComments | worst group accuracy: 70.0% -> 71.8% | Helped |
| Apply ASGDRO rather than ERM to DPLCLIP | DPLCLIP · DomainBed average | average accuracy: 72.6 -> 74.0 | Helped |
| Apply ASGDRO rather than GDRO to DPLCLIP | DPLCLIP · DomainBed average | average accuracy: 72.9 -> 74.0 | Helped |
| Learn diverse invariant features with ASGDRO rather than GDRO | H-CMNIST · TestBed 2: Spu & Shape | accuracy: 57.53 ± 2.11 -> 66.62 ± 5.61 | Helped |
Surprising findings
- ASAM, which uses sharpness awareness, fails to eliminate spurious correlations on subpopulation shift tasks and underperforms GDRO.
Limitations
- Theoretical guarantees rely on linear-model approximations and Taylor expansion arguments.
- CivilComments performance is only comparable to the best baselines rather than best.
Experimental setup
Datasets: CMNIST Waterbirds CelebA CivilComments Camelyon17 FMoW Amazon RxRx1 PACS VLCS OfficeHome TerraIncognita
Benchmarks: H-CMNIST CMNIST Waterbirds CelebA CivilComments WILDS DomainBed
Metrics: average accuracy worst-group accuracy Hessian eigenvalues
Baselines: ERM ASAM GDRO IRM IB-IRM V-REx CORAL DomainMix Fish LISA GSAM RDM
Builds on: Group DRO Sharpness-aware Minimization Adaptive SAM IRM DPLCLIP
Closest papers by meaning
- Distributionally Robust Post-hoc Classifiers under Prior Shifts (ICLR 2023 · similarity 0.89)
- Examining and Combating Spurious Features under Distribution Shift (ICML 2021 · similarity 0.88)
- Tolerant Algorithms for Learning with Arbitrary Covariate Shift (NeurIPS 2024 · similarity 0.87)
- Transformation-Invariant Learning and Theoretical Guarantees for OOD Generalization (NeurIPS 2024 · similarity 0.86)
- Bridging Multicalibration and Out-of-distribution Generalization Beyond Covariate Shift (NeurIPS 2024 · similarity 0.86)
- Towards Robustness to Natural Variations and Distribution Shift (Student Abstract) (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.