Research map / Label-efficient and robust learning / Recognition and long-tail
LaSCal: Label-Shift Calibration without target labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a label-free, consistent class-wise calibration error estimator under label shift and uses it as a loss for post-hoc temperature scaling (LaSCal) to calibrate models on unlabeled shifted target data.
Problem
Calibration error estimation and post-hoc calibration under label shift when no labeled target data is available; existing CE estimators assume covariate shift or require target labels.
Approach
Estimate target conditional expectation E_t[Y_c | z_c] by reweighting source kernel-based ratio estimates with label-shift importance weights (RLLS), yielding a consistent label-free CE estimator that is minimized over temperature to calibrate on the unlabeled target.
Key results
- LaSCal achieves macro-averaged CE of 5.87 on CIFAR-100-LT (IF=10) vs 28.54 for TempScal and 25.20 for CPMCN, and 3.38 on Amazon vs 3.05 for EM-BCTS.
- On iWildCam, LaSCal reaches macro-average CE 14.71 vs 14.67 for TransCal and 19.36 uncalibrated.
- The label-free CE estimator closely tracks ground-truth CE across label-shift intensities on Camelyon17 and across models on Amazon and iWildCam.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| calibrate on unlabeled target data with LaSCal instead of CPCS | ResNet models · CIFAR-100-LT (IF=10) | classwise CE ×100: 25.13 -> 5.87 | Helped |
| calibrate on unlabeled target data with LaSCal instead of EM-BCTS | ResNet models · CIFAR-10-LT (IF=10) | classwise CE ×100: 4.39 -> 4.63 | Hurt |
| calibrate on unlabeled target data with LaSCal instead of EM-BCTS | Transformer models · Amazon Reviews | classwise CE ×100: 3.05 -> 3.38 | Hurt |
| calibrate on unlabeled target data with LaSCal instead of TransCal | Image models · iWildCam | classwise CE ×100: 14.67 -> 14.71 | No clear effect |
| use RLLS importance weights rather than ELSA weights (ablation) | LaSCal CE estimator · CIFAR-100-LT (IF=10), ResNet-20 | estimated classwise CE ×100: 96.61±0.54 -> 63.97±0.28 | Helped |
| use RLLS importance weights rather than ELSA weights (ablation) | LaSCal CE estimator · CIFAR-10-LT (IF=10), ResNet-20 | estimated classwise CE ×100: 9.06±0.31 -> 8.93±0.33 | Helped |
Surprising findings
- On CIFAR-10-LT, EM-BCTS (calibrated on labeled source) gives lower macro CE (4.39) than LaSCal (4.63), so LaSCal is not uniformly best.
- Several weight estimators (BBSL, ELSA, EM-BCTS) produce abnormal CE values on CIFAR-100-LT, while RLLS is stable.
Limitations
- Designed only for label shift; covariate shift is not addressed by the estimator.
- Depends on quality of importance weights from upstream weight estimators, which can be unreliable when classes are under-represented.
- Requires a sufficient number of samples (e.g., ~4000) for accurate CE estimation.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 CIFAR-10-LT CIFAR-100-LT Camelyon17 iWildCam Amazon Reviews
Benchmarks: CIFAR-10-LT CIFAR-100-LT Amazon Reviews iWildCam Camelyon17
Metrics: L2 class-wise calibration error x100 L1 top-label ECE accuracy
Baselines: TempScal CPCS TransCal HeadToTail EM-BCTS CPMCN VectScal EnsTempScal IROvA
Builds on: Temperature scaling RLLS Popordanoska et al. 2022 (consistent CE estimators) BBSL ELSA
Closest papers by meaning
- Beyond One-Hot Labels: Semantic Mixing for Model Calibration (ICML 2025 · similarity 0.87)
- Reassessing How to Compare and Improve the Calibration of Machine Learning Models (ICLR 2025 · similarity 0.86)
- Learning Label Shift Correction for Test-Agnostic Long-Tailed Recognition (ICML 2024 · similarity 0.85)
- Bridging Multicalibration and Out-of-distribution Generalization Beyond Covariate Shift (NeurIPS 2024 · similarity 0.85)
- Testing Calibration in Nearly-Linear Time (NeurIPS 2024 · similarity 0.85)
- Balancing Two Classifiers via A Simplex ETF Structure for Model Calibration (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.