Research map / Label-efficient and robust learning / Recognition and long-tail
Improving self-training under distribution shifts via anchored confidence with theoretical guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AnCon, a self-training method that smooths pseudo labels with an uncertainty-aware temporal ensemble (relative confidence thresholding over past hard predictions), giving theoretical guarantees and 8-16% gains under distribution shifts without extra forward passes.
Problem
Self-training under distribution shifts suffers from noisy pseudo labels because prediction confidence becomes poorly aligned with accuracy; existing fixes (neighborhood, ensemble, augmentation-consistency methods) are computationally expensive.
Approach
Replace one-hot pseudo labels with label-smoothed targets built from a temporal ensemble of past hard predictions, where each past prediction is weighted uniformly only if its confidence exceeds an EMA-based relative threshold; this selective temporal consistency is shown to be asymptotically correct and to reduce the self-training optimality gap.
Key results
- AnCon improves self-training on ImageNet-C by about 16% on average and by 20% and 52% at corruption intensities 4 and 5.
- On Office-31, OfficeHome and VisDa, AnCon reduces average self-training test error by 5%, 6% and 13% respectively, performing comparably to ELR with its tuned coefficient.
- AnCon with NRC reaches 89.07% on Office-31 average vs 87.42% for NRC alone, and yields much lower ECE than other self-training variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add AnCon to self-training | Self-Training · Office-31 | accuracy: 80.47 -> 81.37 | Helped |
| Add AnCon to self-training | Self-Training · OfficeHome | accuracy: 61.55 -> 63.91 | Helped |
| Add AnCon to self-training | Self-Training · VisDa | accuracy: 67.77 -> 71.11 | Helped |
| Add AnCon to self-training instead of ELR | Self-Training · VisDa | accuracy: 71.89 -> 71.11 | Hurt |
| Add AnCon to GCE | GCE · OfficeHome | accuracy: 63.00 -> 64.81 | Helped |
| Add AnCon to NRC | NRC · OfficeHome | accuracy: 63.92 -> 67.96 | Helped |
| Use relative confidence thresholding instead of entropy-based weights (ablation) | AnCon · OfficeHome domain pairs | accuracy | Helped |
| Use relative confidence thresholding instead of max-probability weights (ablation) | AnCon · OfficeHome domain pairs | accuracy | Helped |
| Use hard rather than soft predictions (ablation) | AnCon · OfficeHome domain pairs | accuracy | Helped |
| Vary lambda from the default 0.3 (ablation) | AnCon · OfficeHome domain pairs | accuracy: less than 1% | No clear effect |
| Vary beta from the default 0.9 (ablation) | AnCon · OfficeHome domain pairs | accuracy: less than 1% | No clear effect |
Surprising findings
- Sophisticated uncertainty-based weighting (entropy, maxprob) and soft predictions reduce performance compared with the simple relative thresholding and hard predictions.
Limitations
- The fixed lambda is not adaptive; theory suggests the optimal value depends on temporal ensemble quality.
- The on-average correct-prediction condition can be violated in challenging self-training scenarios.
- Self-training methods still fall short without sophisticated model designs tailored for distribution shifts.
Experimental setup
Datasets: Office-31 OfficeHome VisDa ImageNet-C
Benchmarks: Office-31 OfficeHome VisDa ImageNet-C
Metrics: test accuracy expected calibration error average test error reduction
Baselines: Self-Training ELR GCE NRC
Builds on: early learning regularization label smoothing neighborhood reciprocity clustering generalized cross-entropy
Closest papers by meaning
- Label Shift Meets Online Learning: Ensuring Consistent Adaptation with Universal Dynamic Regret (CVPR 2025 · similarity 0.86)
- Improved Techniques for Training Consistency Models (ICLR 2024 · similarity 0.84)
- Improved Training Technique for Latent Consistency Models (ICLR 2025 · similarity 0.84)
- Adaptive Estimation and Learning under Temporal Distribution Shift (ICML 2025 · similarity 0.84)
- Improved Self-Training for Test-Time Adaptation (CVPR 2024 · similarity 0.84)
- Learning from Noisy Labels via Conditional Distributionally Robust Optimization (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.