Research map / Label-efficient and robust learning / Recognition and long-tail
Optimizing importance weighting in the presence of sub-population shifts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes optimizing importance weights via a bi-level procedure that accounts for finite-sample variance, and applies it to last-layer retraining for sub-population shifts, improving GW-ERM, SUBG, DFR, GDRO and JTT.
Problem
Heuristic importance weights (e.g., likelihood ratios for group-weighted ERM) are suboptimal under finite training samples because they inflate estimator variance, hurting generalization under sub-population shift.
Approach
Treat importance weights as hyperparameters chosen to minimize a validation loss in a bi-level optimization (weights and model parameters optimized jointly), with hyper-gradients computed via the implicit function theorem; the bias-variance trade-off is shown analytically for linear regression.
Key results
- Optimized weights improved the targeted objective in 12 of 15 method-dataset cases at 10% significance, with a maximum gain of 5.61%.
- Optimizing weights improved the non-targeted metric in many cases, with more than 1% gains in 10 cases (maximum 4.03%).
- Benefit of optimized weights is largest when the minority groups or total sample size are small (e.g., Waterbirds, CelebA subsampled to around 10%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| optimize group weights instead of using each method's standard weights | GW-ERM, SUBG, DFR, GDRO, and JTT · Waterbirds, CelebA, and MultiNLI | method objective accuracy: maximum of 5.61% | Helped |
| optimize group weights instead of using standard weights | GW-ERM, SUBG, DFR, GDRO, and JTT · Waterbirds, CelebA, and MultiNLI | non-objective accuracy: maximum of 4.03% | Mixed |
| optimize group weights instead of using grid-searched standard weights | JTT · CelebA | worst-group accuracy | No clear effect |
| optimize group weights instead of standard weights | GW-ERM · CelebA, using fractions of the original training and validation data | weighted average accuracy: 10%; 5% | Mixed |
| optimize group weights instead of standard weights | GW-ERM · Waterbirds | weighted average accuracy | Helped |
| optimize group weights instead of standard weights | GW-ERM · Waterbirds | worst-group accuracy | Helped |
Surprising findings
- Optimized weights often also improved the metric they were not optimized for, e.g., worst-group accuracy for GW-ERM, SUBG and DFR, although weighted average accuracy and group robustness are often opposed.
Limitations
- JTT on CelebA did not improve even on validation set, suggesting the bi-level procedure failed to find better weights there.
- No formal convergence guarantees; the weight optimization may be non-convex.
- Hyper-gradient computation requires refitting the model and inverting Hessians, which is costly for full network retraining.
Experimental setup
Datasets: Waterbirds CelebA MultiNLI
Benchmarks: Waterbirds CelebA MultiNLI
Metrics: weighted average accuracy worst-group accuracy
Baselines: GW-ERM SUBG DFR GDRO JTT
Builds on: group-weighted empirical risk minimization subsampling large groups group distributionally robust optimization Just Train Twice deep feature reweighting
Closest papers by meaning
- Thumb on the Scale: Optimal Loss Weighting in Last Layer Retraining (NeurIPS 2025 · similarity 0.87)
- Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks (NeurIPS 2025 · similarity 0.86)
- Variable Importance in High-Dimensional Settings Requires Grouping (AAAI 2024 · similarity 0.85)
- Pay Attention to Small Weights (NeurIPS 2025 · similarity 0.85)
- Initializing Models with Larger Ones (ICLR 2024 · similarity 0.85)
- Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.