Research map / Label-efficient and robust learning / Domain adaptation and generalization

Pseudo-Calibration: Improving Predictive Uncertainty Estimation in Unsupervised Domain Adaptation

ICML 2024 · Label-efficient and robust learning › Domain adaptation and generalization · approach: Feature alignment for unsupervised adaptation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PseudoCal, a post-hoc calibration method for unsupervised domain adaptation that uses inference-stage mixup on unlabeled target data to build a labeled pseudo-target set, then fits temperature scaling on it.

Problem

Predictive uncertainty of unsupervised domain adaptation models is poorly calibrated on the unlabeled target domain, and existing calibration methods rely on covariate-shift assumptions, extra training, or labeled source data.

Approach

Treat UDA calibration as an unsupervised target-domain problem: mix cross-cluster target samples at inference time with a fixed mix ratio (0.65) and their pseudo-labels to synthesize a labeled pseudo-target set with correct/wrong statistics resembling the real target set, then apply temperature scaling on it.

Key results

• PseudoCal achieves average ECE of about 5.2% across UDA settings versus 10.3% for TransCal and 10.3% for TempScal-src, approaching the oracle ECE of 2.5%.
• On Office-Home closed-set UDA, PseudoCal improves average ECE by 4.33% over the second-best calibration method.
• On source-free UDA, PseudoCal reduces ECE by 7.44% on DomainNet and 15.05% on Image-Sketch relative to Ensemble.

Measured effects reported in the paper

Surprising findings

• PseudoCal can increase ECE when the initial ECE is already small (e.g., D target in Office-31).
• Inference-stage mixup with lambda near 0.5 or 1.0 degrades performance, and PseudoCal becomes ineffective when UDA model accuracy is below about 15%.

Limitations

• May increase ECE when initial calibration error is already small.
• Fails with very few unlabeled target samples or when pseudo-label accuracy is very low (near random).
• Depends on the cluster assumption and the mix ratio lambda must be in a moderate range (0.6-0.8).

Experimental setup

Datasets: Office-31 Office-Home VisDA DomainNet ImageNet-Sketch GTA5 SYNTHIA Cityscapes

Benchmarks: Office-31 Office-Home VisDA DomainNet Image-Sketch Cityscapes segmentation (GTA5, SYNTHIA sources)

Metrics: ECE NLL Brier score

Baselines: No Calib. TempScal-src CPCS TransCal Ensemble MixUp-Beta CutMix Pseudo-Label Filtered-PL

Builds on: temperature scaling mixup cluster assumption TransCal CPCS

Closest papers by meaning

• Transferable Calibration with Lower Bias and Variance in Domain Adaptation (NeurIPS 2020 · similarity 0.90)
• Guiding Pseudo-Labels With Uncertainty Estimation for Source-Free Unsupervised Domain Adaptation (CVPR 2023 · similarity 0.90)
• Towards Reliable Model Selection for Unsupervised Domain Adaptation: An Empirical Study and A Certified Baseline (NeurIPS 2024 · similarity 0.89)
• Make the U in UDA Matter: Invariant Consistency Learning for Unsupervised Domain Adaptation (NeurIPS 2023 · similarity 0.89)
• Pseudo-Labeling for Unsupervised Domain Adaptation with Kernel GLMs (NeurIPS 2026 · similarity 0.88)
• Can We Evaluate Domain Adaptation Models Without Target-Domain Labels? (ICLR 2024 · similarity 0.87)