Research map / Learning theory and algorithms / Prediction and decision losses
Boosting Test Performance with Importance Sampling--a Subpopulation Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a dataset bias analysis (DBA) framework grounded in importance sampling that yields a closed-form weight for subpopulation shift, with three estimators (DBCM) for known/unknown attributes, improving average and worst-group accuracy on Waterbirds, ColorMNIST and CivilComments.
Problem
Subpopulation shift from spurious correlations degrades test performance; existing debiasing methods raise worst-group accuracy but lower average accuracy below ERM, and lack a unified statistical explanation.
Approach
Frames model selection and training under subpopulation shift as importance sampling with weight p(x,y|I_te)/p(x,y|I_tr), derives a closed-form weight under stated assumptions (uniform attribute, group split m), and estimates the one unknown term p(s=y|y,x,I_tr) either from known attributes or via overfitted models acting as a bias corrector.
Key results
- With known attributes, DBCM reaches 96.67% average accuracy on ColorMNIST (0.5%) vs 81.69% for ERM, and 94.01% average on Waterbirds vs 88.25% for ERM.
- Without attributes, DBCM (D_tr ~ D_va) achieves 94.63% average on ColorMNIST (0.5%) and 91.04% average on Waterbirds (D_tr not~ D_va variant), the highest average among compared methods.
- DBCM is reported as the only method that consistently outperforms ERM on average accuracy across the datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply importance-sampling reweighting using known attribute s | DBCM versus ERM · ColorMNIST (0.5%), known s | average accuracy: 81.69 ± 0.10 -> 96.67 ± 0.27 | Helped |
| Apply importance-sampling reweighting using known attribute s | DBCM versus ERM · ColorMNIST (2%), known s | average accuracy: 95.23 ± 0.07 -> 98.76 ± 0.20 | Helped |
| Apply importance-sampling reweighting using known attribute s | DBCM versus ERM · Waterbirds, known s | average accuracy: 88.25 ± 0.16 -> 94.01 ± 0.19 | Helped |
| Apply importance-sampling reweighting using known attribute s | DBCM versus ERM · CivilComments, known s | average accuracy: 87.59 ± 0.38 -> 87.85 ± 0.15 | Helped |
| Estimate the bias from train/validation predictive-model differences | DBCM (train ~ validation) versus ERM · ColorMNIST (0.5%), unknown s | average accuracy: 81.69 ± 0.10 -> 94.63 ± 0.35 | Helped |
| Estimate the bias from train/validation predictive-model differences | DBCM (train ~ validation) versus ERM · Waterbirds, unknown s | average accuracy: 88.25 ± 0.16 -> 88.25 ± 0.05 | No clear effect |
| Estimate the bias from training predictive confidence | DBCM (train not ~ validation) versus ERM · ColorMNIST (0.5%), unknown s | worst-group accuracy: 1.14 ± 0.40 -> 3.41 ± 0.70 | Helped |
| Estimate the bias from training predictive confidence | DBCM (train not ~ validation) versus ERM · Waterbirds, unknown s | average accuracy: 88.25 ± 0.16 -> 91.04 ± 0.07 | Helped |
| Use known attribute s for correction | DBCM versus ERM · CivilComments, known s | worst-group accuracy: 48.17 ± 2.61 -> 43.33 ± 2.15 | Hurt |
| Estimate bias from train/validation predictive-model differences | DBCM (train ~ validation) versus ERM · CivilComments, unknown s | worst-group accuracy: 48.17 ± 2.61 -> 43.41 ± 2.20 | Hurt |
| Use the training predictive-confidence proxy for bias | DBCM (train not ~ validation) versus ERM · ColorMNIST (0.5%), unknown s | average accuracy: 81.69 ± 0.10 -> 86.12 ± 0.29 | Helped |
Surprising findings
- Most subpopulation debiasing methods lose average accuracy relative to ERM; the authors attribute this to a misspecified test distribution in the weight function.
- Simple reweighting with a closed-form weight can match or exceed more complex methods on worst-group accuracy in some settings.
Limitations
- Relies on several strong assumptions (uniform spurious attribute, identical p(x|y,s) across train and test, a two-group split), some only partly relaxed in the appendix.
- No theoretical guarantee on worst-group accuracy; worst-group gains are not quantified by theory.
- Worst-group accuracy is not best on CivilComments and some ColorMNIST settings.
Experimental setup
Datasets: Waterbirds ColorMNIST CivilComments
Benchmarks: Waterbirds ColorMNIST CivilComments
Metrics: average accuracy worst-group accuracy
Baselines: ERM Mixup LISA JTT Focal Loss GroupDRO MMD ReSample ReWeight
Builds on: empirical risk minimization importance sampling JTT LISA GroupDRO
Closest papers by meaning
- Learning With Multi-Group Guarantees For Clusterable Subpopulations (ICML 2025 · similarity 0.87)
- Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods (AAAI 2024 · similarity 0.85)
- Sample-Optimal Agnostic Boosting with Unlabeled Data (ICML 2025 · similarity 0.85)
- Prediction-Powered Active Testing (NeurIPS 2026 · similarity 0.85)
- Diverse Prototypical Ensembles Improve Robustness to Subpopulation Shift (ICML 2025 · similarity 0.85)
- Fairness without Harm: An Influence-Guided Active Sampling Approach (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.