Research map / Label-efficient and robust learning / Recognition and long-tail
The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how class-balancing (subsetting, upsampling, upweighting), a proposed mixture balancing method, model scaling, and spectral imbalance of group covariances affect worst-group accuracy when finetuning pretrained vision and language models under spurious correlations.
Problem
Worst-group accuracy under spurious correlations depends on subtle details of finetuning, including class-balancing technique and model scale, which are not well understood for pretrained models.
Approach
Shows that upsampling and upweighting cause catastrophic WGA collapse with long training while subsetting can help, proposes a mixture balancing method (random subset to a chosen class-imbalance ratio, then upsampling), finds that scaling pretrained models helps WGA only with appropriate class-balancing, and introduces the intra-class spectral norm ratio to show minority group covariances have lar
Key results
- Upsampling and upweighting experience catastrophic WGA collapse over long training on CelebA and CivilComments, while subsetting harms Waterbirds WGA due to a small minority group within the majority class.
- Mixture balancing outperforms subsetting, upsampling, and upweighting on Waterbirds and CivilComments, and mitigates the collapse on CelebA.
- Scaling class-balanced pretrained models improves WGA by up to 12% for interpolating models and 40% for non-interpolating models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use mixture balancing instead of subsetting, upsampling, or upweighting | ERM finetuning · Waterbirds | worst-group accuracy | Helped |
| Use mixture balancing instead of subsetting, upsampling, or upweighting | ERM finetuning · CivilComments | worst-group accuracy | Helped |
| Use mixture balancing instead of upsampling or upweighting | ERM finetuning · CelebA | worst-group accuracy | Mixed |
| Remove data to create a class-balanced subset | ERM finetuning · Waterbirds | worst-group accuracy | Hurt |
| Use class-balanced subsetting | ERM finetuning · CelebA | worst-group accuracy | Helped |
| Use class-balanced upsampling or upweighting for long training runs | ERM finetuning · CelebA and CivilComments | worst-group accuracy | Hurt |
| Add subsetting before adjusting sampling probabilities (ablation) | Mixture balancing · Waterbirds, CelebA, and CivilComments | worst-group accuracy | Helped |
| Vary the class-imbalance ratio from the subsetting endpoint to the upsampling endpoint (ablation) | Mixture balancing · Waterbirds, CelebA, and CivilComments | worst-group accuracy | Mixed |
| Scale pretrained models while using appropriate class-balancing | Pretrained model finetuning · Waterbirds, CelebA, and CivilComments | worst-group accuracy: up to 12%; 40% | Mixed |
| Scale models on class-imbalanced data or with an inappropriate balancing method | Pretrained model finetuning · CivilComments | worst-group accuracy | Hurt |
| Condition covariance comparisons on class | Finetuned group representations · Waterbirds, CelebA, CivilComments, and MultiNLI | intra-class spectral norm ratio | Helped |
Surprising findings
- Class-balanced upsampling and upweighting can yield WGA no better than no class-balancing when trained to convergence with standard hyperparameters, despite prior reports of their benefit.
- The group with the largest top eigenvalue is a minority group but not necessarily the worst-accuracy group; minority eigenvalues are larger than majority ones only when conditioned on the class.
Limitations
- Mixture balancing's optimal class-imbalance ratio requires a group-annotated validation set, though worst-class accuracy and bias-unsupervised score were shown to be alternatives.
- Methods exploit the structure of spurious correlations and may not transfer to complete correlation settings or general out-of-distribution generalization.
- Theoretical explanation of the collapse and the spectral findings is left to future work.
Experimental setup
Datasets: Waterbirds CelebA CivilComments MultiNLI ImageNet-1K Book Corpus English Wikipedia
Benchmarks: Waterbirds CelebA CivilComments MultiNLI
Metrics: worst-group accuracy average accuracy intra-class spectral norm ratio
Baselines: ERM subsetting upsampling upweighting Just Train Twice worst-class accuracy validation bias-unsupervised validation score
Builds on: Just Train Twice empirical risk minimization AdamW
Closest papers by meaning
- Subgroups Matter for Robust Bias Mitigation (ICML 2025 · similarity 0.87)
- Balanced Data, Imbalanced Spectra: Unveiling Class Disparities with Spectral Imbalance (ICML 2024 · similarity 0.87)
- Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference (ICML 2024 · similarity 0.86)
- Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustness (CVPR 2025 · similarity 0.86)
- Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations (CVPR 2024 · similarity 0.86)
- Label-Efficient Group Robustness via Out-of-Distribution Concept Curation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.