Research map / Label-efficient and robust learning / Domain adaptation and generalization
IW-GAE: Importance weighted group accuracy estimation for improved calibration and model selection in unsupervised domain adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IW-GAE, which estimates group accuracy under covariate shift by learning binned importance weights via a constrained optimization that matches source Monte-Carlo and IW-based group accuracy estimates, improving calibration and model selection in UDA.
Problem
Model calibration and model selection in unsupervised domain adaptation are difficult because distribution shift breaks i.i.d. assumptions and target labels are unavailable.
Approach
Group test samples by softmax confidence and estimate per-group target accuracy; learn binned importance weights (within Clopper-Pearson confidence intervals) by minimizing the discrepancy between an IW-based and a Monte-Carlo source group accuracy estimate, so that the optimal IW yields an accurate target group accuracy estimator.
Key results
- IW-GAE reaches average ECE of 8.93 on OfficeHome (MDD) versus 19.79 for the best prior baseline CPCS, and 6.32 on DomainNet (CDAN).
- In checkpoint selection on OfficeHome (MDD), IW-GAE achieves average accuracy 68.48 versus 68.38 for InfoMax/TransScore and 67.47 for vanilla.
- The authors report average improvements of 22% in calibration and 14% in model selection over state-of-the-art.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IW-GAE instead of CPCS | MDD with ResNet-50 · OfficeHome (all domain pairs) | ECE: 19.79 -> 8.93 | Helped |
| Use IW-GAE instead of TS | CDAN with ResNet-50 · DomainNet (all domain pairs) | ECE: 9.22 -> 6.32 | Helped |
| Use IW-GAE instead of Vanilla | CDAN · VisDA-2017 (Sim-Rw) | ECE: 21.63 -> 14.70 | Helped |
| Select with IW-GAE instead of TransScore | MDD, ResNet-50 · OfficeHome hyperparameter selection (all domain pairs) | test accuracy: 67.87 -> 67.95 | Helped |
| Select with IW-GAE instead of TransScore | MDD, ResNet-50 · OfficeHome checkpoint selection (all domain pairs) | test accuracy: 68.38 -> 68.48 | Helped |
| Group predictions by importance weight instead of maximum softmax output (ablation) | IW-GAE with MDD, ResNet-50 · OfficeHome (four selected domain pairs) | ECE: 9.97 -> 22.31 | Hurt |
| Replace the confidence-interval constraints with the interval [1/6, 6.0] (ablation) | IW-GAE with MDD, ResNet-50 · OfficeHome (four selected domain pairs) | ECE: 9.97 -> 16.82 | Hurt |
| Use CI midpoints as importance weights instead of optimizing the weights (ablation) | IW-GAE with MDD, ResNet-50 · OfficeHome (four selected domain pairs) | ECE: 9.97 -> 24.70 | Hurt |
Surprising findings
- IW-Mid, a simple midpoint of the importance weight confidence interval, sometimes beats other IW baselines but is unstable and can underperform vanilla.
- All IW-based methods fail to improve over the standard approach in the i.i.d. setting with pretrained large language models (XLM-R, GPT-2).
Limitations
- IW-based methods do not help in i.i.d. or mild-shift settings such as pretrained LLMs.
- Performance drops without the confidence-interval constraints (W/O CI variant), so the method depends on a good CI estimator.
- Theoretical bounds rely on assumptions (identical accuracy assumption, covariate shift only) that may not hold exactly.
Experimental setup
Datasets: OfficeHome VisDA-2017 DomainNet
Benchmarks: OfficeHome DomainNet VisDA-2017
Metrics: expected calibration error (ECE) mean test accuracy of selected checkpoint/hyperparameter
Baselines: Vanilla TS CPCS IW-TS TransCal PTS AvUTS IW-Mid IWCV DEV InfoMax TransScore
Builds on: temperature scaling importance weighted cross validation TransCal Clopper-Pearson confidence interval MDD
Closest papers by meaning
- Pseudo-Labeling for Unsupervised Domain Adaptation with Kernel GLMs (NeurIPS 2026 · similarity 0.86)
- Towards Reliable Model Selection for Unsupervised Domain Adaptation: An Empirical Study and A Certified Baseline (NeurIPS 2024 · similarity 0.85)
- Group-robust Sample Reweighting for Subpopulation Shifts via Influence Functions (ICLR 2025 · similarity 0.85)
- Pseudo-Calibration: Improving Predictive Uncertainty Estimation in Unsupervised Domain Adaptation (ICML 2024 · similarity 0.84)
- Rank-Constrained Adaptation for Reliable Real-World Performance (NeurIPS 2026 · similarity 0.84)
- The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.