Research map / Learning theory and algorithms / Sample complexity
Evaluating model performance under worst-case subpopulations
Mike Li, Hongseok Namkoong, Shangzhou Xia
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune zero-shot CLIP on FMoW | CLIP · FMoW ID val | accuracy: 19.3% -> 70.2% | Improved |
| fine-tune zero-shot CLIP on FMoW | CLIP · FMoW OOD robustness | relative robustness | Hurt |
| weight-space ensemble of zero-shot and fine-tuned CLIP (WiSE-FT, lambda=0.4) | CLIP fine-tuned · FMoW OOD test | accuracy: 0.53 -> 0.56 | Improved |
| CLIP WiSE-FT versus DenseNet ERM | FMoW models · FMoW OOD Africa | cross-entropy loss: 5.41 -> 3.08 | Improved |
| CLIP WiSE-FT versus DenseNet ERM | FMoW models · FMoW OOD Africa | accuracy: 0.33 -> 0.38 | Improved |
| CLIP WiSE-FT versus DenseNet ERM | FMoW models · FMoW OOD test | cross-entropy loss: 3.50 -> 2.84 | Improved |
| DenseNet IRM versus DenseNet ERM | DenseNet · FMoW ID val | cross-entropy loss: 2.78 -> 2.44 | Improved |
| DenseNet IRM versus DenseNet ERM | DenseNet · FMoW OOD test | accuracy: 0.53 -> 0.51 | Hurt |
| DPN-68 versus DenseNet ERM | FMoW models · FMoW OOD Africa | cross-entropy loss: 5.41 -> 5.61 | Hurt |
| include label Y in subpopulation definition Z | worst-case estimator W_alpha · FMoW ID val | worst-case loss | Hurt |
| reduce subpopulation size alpha | ImageNet pretrained models vs CLIP WiSE-FT · FMoW ID val | worst-case loss gap | Hurt |
| evaluate linear regression vs expressive models on worst-case subpopulations | linear model · Warfarin dosing | squared error | No clear effect |
Closest papers by meaning
- Multigroup Robustness (ICML 2024 · similarity 0.87)
- Achievable distributional robustness when the robust risk is only partially identified (NeurIPS 2024 · similarity 0.87)
- Learning With Multi-Group Guarantees For Clusterable Subpopulations (ICML 2025 · similarity 0.85)
- All-Purpose Mean Estimation over R: Optimal Sub-Gaussianity with Outlier Robustness and Low Moments Performance (ICML 2025 · similarity 0.85)
- Foundation Model-oriented Robustness: Robust Image Model Evaluation with Pretrained Models (ICLR 2024 · similarity 0.84)
- Boosting Test Performance with Importance Sampling--a Subpopulation Perspective (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
