Research map / Label-efficient and robust learning / Domain adaptation and generalization
Can We Evaluate Domain Adaptation Models Without Target-Domain Labels?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Transfer Score, an unsupervised metric combining classifier angular uniformity, Hopkins statistic, and mutual information to evaluate, select, tune, and checkpoint UDA models without target labels.
Problem
Evaluating and selecting unsupervised domain adaptation (UDA) models is infeasible in practice because target-domain labels are unavailable, and UDA training can cause negative transfer and overfitting.
Approach
A label-free Transfer Score combines classifier spatial uniformity (angles between classifier weight vectors vs. ideal simplex angle), target-feature clustering tendency (Hopkins statistic), and normalized mutual information of target predictions; a saturation level (coefficient of variation over a sliding window) picks the checkpoint.
Key results
- Transfer Score reaches a correlation of 0.97 with target-domain accuracy on DANN Office-31 A→W, versus 0.75 for MMD and 0.65 for PAD.
- On Office-Home, Ours selects the best UDA method and beats ATC on the tested tasks (e.g., W→A 73.3 for CDAN-based selection vs. 70.7 for ATC).
- Checkpoint selection improves FixMatch on VisDA-17 by 17.4 points (49.2 to 66.6) and SAFN on DomainNet (c→p) by about 11.4 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select the checkpoint using Transfer Score rather than using the last epoch | UDA models on VisDA-17 and DomainNet · VisDA-17 and DomainNet c→p/p→c | accuracy: 66.9±0.4 -> 68.3±0.3 (+1.4); 29.6±5.8 -> 41.0±0.5 (+11.4); 42.9±0.2 -> 42.5±0.1 (-0.4) | Mixed |
| select the checkpoint using Transfer Score rather than the last epoch | SHOT, CAN, CST, and AaD · VisDA-17 | accuracy: 77.5 -> 78.0 (0.5); 86.4 -> 86.8 (0.4); 72.0 -> 83.8 (11.8); 83.4 -> 85.9 (2.5) | Helped |
| use Transfer Score checkpoint selection | FDA on GTA-5→Cityscapes · GTA-5→Cityscapes | mIoU: 37.7 -> 39.7 | Helped |
| set MCC temperature to 3 instead of 0.1, 1, 9, or 27 | MCC · VisDA-17 | accuracy: 55.5 -> 76.5; 71.7 -> 76.5; 56.1 -> 76.5; 51.0 -> 76.5 | Helped |
| omit uniformity from the full Transfer Score (ablation) | MCC checkpoint selection · DomainNet c→p | accuracy: 37.3 -> 36.5 | Hurt |
| omit Hopkins statistic from the full Transfer Score (ablation) | MCC checkpoint selection · DomainNet c→p | accuracy: 37.3 -> 36.5 | Hurt |
| omit mutual information from the full Transfer Score (ablation) | MCC checkpoint selection · DomainNet c→p | accuracy: 37.3 -> 36.0 | Hurt |
| use the full Transfer Score as a target-domain regularizer instead of source-only training | ResNet-50 UDA · Office-31 A→W and Office-Home Ar→Cl | accuracy: 68.4 -> 92.5; 34.9 -> 47.6 | Helped |
Surprising findings
- Existing metrics such as MMD and PAD are poorly indicative of target accuracy and do not reveal negative transfer.
- Using the Transfer Score as a direct training regularizer improved A→W accuracy substantially in a supplementary experiment (68.4 to 92.5), though this is partially self-referential because TS components are optimized.
Limitations
- Metric may not pick the absolute best model when performance differences are very small.
- Methods that directly optimize a TS component (e.g., SAFN clustering, MI-regularized methods) can make the metric less reliable.
- Checkpoint selection can fail to find the best epoch when accuracy keeps increasing (e.g., MDD).
Experimental setup
Datasets: Office-31 Office-Home VisDA-17 DomainNet GTA-5 Cityscapes
Benchmarks: Office-31 Office-Home VisDA-17 DomainNet GTA-5 to Cityscapes
Metrics: target-domain accuracy correlation with target accuracy accuracy improvement from checkpoint selection mIoU
Baselines: DEV C-Entropy SND ATC MMD Proxy A-distance
Builds on: MCC DIRT-T SND C-Entropy
Closest papers by meaning
- Make the U in UDA Matter: Invariant Consistency Learning for Unsupervised Domain Adaptation (NeurIPS 2023 · similarity 0.93)
- Towards Reliable Model Selection for Unsupervised Domain Adaptation: An Empirical Study and A Certified Baseline (NeurIPS 2024 · similarity 0.92)
- Distributionally Robust Classification for Multi-source Unsupervised Domain Adaptation (ICLR 2026 · similarity 0.92)
- Unsupervised Multi-Source Domain Adaptation Without Access to Source Data (CVPR 2021 · similarity 0.92)
- CASUAL: Conditional Support Alignment for Domain Adaptation with Label Shift (AAAI 2025 · similarity 0.91)
- Toward Improving Robustness and Accuracy in Unsupervised Domain Adaptation (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.