Research map / Label-efficient and robust learning / Test-time adaptation
BREEDS: Benchmarks for Subpopulation Shift
Shibani Santurkar, Dimitris Tsipras, Aleksander Madry
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with ERM on source subpopulations and test on unseen target subpopulations | ImageNet classifiers (AlexNet to DenseNet-121) · BREEDS tasks | target accuracy | Hurt |
| Use more accurate source models | standard-trained models · BREEDS tasks | fraction of source accuracy preserved | Improved |
| Retrain last linear layer on target-domain data | source-trained models · BREEDS tasks | target accuracy | Improved |
| Adversarial training (l2 PGD) | ImageNet classifiers · BREEDS tasks | relative accuracy (target/source) | Improved |
| Erase noise augmentation | ImageNet classifiers · BREEDS tasks | relative accuracy (target/source) | Improved |
| Gaussian noise augmentation | ImageNet classifiers · BREEDS tasks | relative accuracy (target/source) | No clear effect |
| Stylized ImageNet training | ImageNet classifiers · BREEDS tasks | target accuracy after last-layer retraining | Hurt |
| Adversarial training followed by last-layer retraining on target | ImageNet classifiers · Living-17 | target accuracy after retraining | Improved |
| Use BREEDS-modified WordNet hierarchy instead of raw WordNet superclasses (ablation) | task construction · human annotator binary tasks | annotator accuracy | Improved |
| Compare model (DenseNet-121) with humans on binary BREEDS tasks | DenseNet-121 · binary BREEDS target domain | accuracy | Hurt |
| Introduce subpopulation shift for human annotators (source to target) | human annotators · BREEDS binary tasks | annotator accuracy | No clear effect |
Closest papers by meaning
- Change is Hard: A Closer Look at Subpopulation Shift (ICML 2023 · similarity 0.90)
- Benchmarking Low-Shot Robustness to Natural Distribution Shifts (ICCV 2023 · similarity 0.90)
- WILDS: A Benchmark of in-the-Wild Distribution Shifts (ICML 2021 · similarity 0.88)
- Diverse Prototypical Ensembles Improve Robustness to Subpopulation Shift (ICML 2025 · similarity 0.86)
- Learning With Multi-Group Guarantees For Clusterable Subpopulations (ICML 2025 · similarity 0.85)
- CODA: A Correlation-Oriented Disentanglement and Augmentation Modeling Scheme for Better Resisting Subpopulation Shifts (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
