Research map / Label-efficient and robust learning / Domain adaptation and generalization
Towards Reliable Model Selection for Unsupervised Domain Adaptation: An Empirical Study and A Certified Baseline
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study of 8 unsupervised model selection methods for unsupervised domain adaptation across closed, partial, open-partial and source-free settings, plus EnsV, an ensemble-as-role-model selector that provably avoids worst-case selection.
Problem
Selecting hyperparameters/checkpoints for unsupervised domain adaptation (UDA) without target labels, in a way that reliably avoids worst-case selections across UDA methods and scenarios.
Approach
Use the prediction-averaged ensemble of all candidate models as a role model for target data, then select the candidate whose predictions best agree with it; a proposition shows the ensemble loss is below the average loss, which is below the worst model's loss, so the selection avoids the worst candidate.
Key results
- EnsV has 0 worst-case selections across 131 closed-set tasks, 24 partial-set tasks, and 17 source-free tasks, while the best baselines (e.g., Entropy, InfoMax, SND, Corr-C, RV) make 2 to 80 worst-case selections depending on setting.
- On Office-Home CDA with six UDA methods, EnsV averages 70.36% selected accuracy versus a best achievable of 70.72% and worst of 60.26%.
- On the 28-candidate two-hyperparameter Office-Home MDD/MCC experiment, EnsV-P achieves 72.26 average vs. 72.86 best and 51.83 worst.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use prediction-averaged ensemble as a role model for selection | UDA model selection · Closed-set UDA tasks | worst-case selections: 0 / 131 | Helped |
| use prediction-averaged ensemble as a role model for selection | UDA model selection · Partial-set UDA tasks | worst-case selections: 0 / 24 | Helped |
| use prediction-averaged ensemble as a role model for selection | UDA model selection · Source-free UDA tasks | worst-case selections: 0 / 17 | Helped |
| select by similarity to the prediction ensemble | CDA model selection · Office-Home | accuracy (%): EnsV 70.36; RV 69.68 | Helped |
| select by similarity to the prediction ensemble | PDA model selection · Office-Home | accuracy (%): EnsV 69.19; DEV 66.67 | Helped |
| select by similarity to the prediction ensemble | OPDA model selection · Office-Home | H-score (%): EnsV 68.96; Entropy 34.27 | Helped |
| select by similarity to the prediction ensemble | SFUDA model selection · Office-31 | accuracy (%): EnsV 87.15; Corr-C 84.18 | Helped |
| select by similarity to the prediction ensemble | UDA model selection · GTAV → Cityscapes | mIoU (%): EnsV 40.69; SND 40.02 | Helped |
| use parameter weight-based rather than prediction-based ensemble as role model (ablation) | EnsV · Office-Home CDA | accuracy (%): EnsV-W 71.72; EnsV 71.74 | Mixed |
| use parameter weight-based rather than prediction-based ensemble as role model (ablation) | EnsV · Office-Home PDA | accuracy (%): EnsV-W 68.21; EnsV 68.06 | Mixed |
Surprising findings
- None of the eight existing model selection methods, including the source-based ones, reliably avoids worst-case selection across UDA methods and settings.
- Corr-C makes worst-case selections on 80 of 131 closed-set tasks, despite being a recent method.
Limitations
- EnsV does not guarantee peak selection and can be suboptimal when the ensemble itself is poor, e.g., when most candidates are poor.
- EnsV struggles to distinguish candidates when performance differences among them are small.
- EnsV may struggle with a pool containing only one good model and one poor model.
Experimental setup
Datasets: Office-31 Office-Home VisDA DomainNet-126 GTAV Cityscapes
Benchmarks: Office-31 Office-Home VisDA DomainNet-126 GTAV to Cityscapes
Metrics: validation accuracy H-score mIoU worst-case selection count
Baselines: SourceRisk IWCV DEV RV Entropy InfoMax SND Corr-C
Builds on: Transfer Learning Library Reverse Validation Soft Neighborhood Density Corr-C Deep Embedded Validation
Closest papers by meaning
- Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation (ICML 2019 · similarity 0.93)
- Can We Evaluate Domain Adaptation Models Without Target-Domain Labels? (ICLR 2024 · similarity 0.92)
- Pseudo-Calibration: Improving Predictive Uncertainty Estimation in Unsupervised Domain Adaptation (ICML 2024 · similarity 0.89)
- Gradient-Based Sample Selection for Black-Box Universal Domain Adaptation (AAAI 2025 · similarity 0.88)
- Universal Semi-Supervised Domain Adaptation by Mitigating Common-Class Bias (CVPR 2024 · similarity 0.87)
- Revisiting Source-Free Domain Adaptation: Insights into Representativeness, Generalization, and Variety (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.