Research map / Label-efficient and robust learning / Recognition and long-tail
On the Limitations of Temperature Scaling for Distributions with Overlaps
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically and empirically that temperature scaling of interpolating ERM models fails on data distributions with class overlaps, while (d-)Mixup training yields good calibration.
Problem
Temperature scaling, a post-training calibration method, is often outperformed by training-time modifications; the paper asks under what data conditions temperature scaling provably fails to calibrate while training-time methods succeed.
Approach
For ERM interpolators on distributions with class-support overlaps, even oracle temperature scaling cannot calibrate (error stays bounded below as overlap grows, and becomes random-like with many classes), whereas a generalized d-Mixup objective mixing d+1 points yields provably bounded calibration error.
Key results
- On 2-class Gaussian data with mu=0.01*1, ERM+TS has NLL 4.2977 and ECE 0.4686 versus 4-Mixup NLL 0.7444 and ECE 0.1018.
- On CIFAR-10 with 50% label noise, ERM+TS NLL is 5.9533 versus Mixup NLL 1.7729.
- On SVHN with 50% label noise, ERM+TS NLL is 4.1807 versus Mixup NLL 1.5030, though ERM has better test error than Mixup on SVHN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ERM + temperature scaling with Mixup training | ResNeXt-50 · 2-class Gaussian, μ = 0.05 | NLL: 3.7331 $\pm 0.4986$ -> 0.7487 $\pm 0.0356$ | Helped |
| Replace ERM + temperature scaling with Mixup training | ResNeXt-50 · CIFAR-10, CIFAR-100, and SVHN with 0%, 25%, and 50% label noise | NLL: CIFAR-10: 2.0109 $\pm 0.1750$ -> \textbf{0.8190} $\pm 0.0349$; 3.2091 $\pm 0.2042$ -> \textbf{1.3102} $\pm 0.0311$; 5.95 | Helped |
| Replace ERM + temperature scaling with Mixup training | ResNeXt-50 · CIFAR-100 and SVHN with 0%, 25%, and 50% label noise | NLL: CIFAR-100: 5.4969 $\pm 0.8198$ -> \textbf{2.4608} $\pm 0.0482$; 6.3767 $\pm 0.4089$ -> \textbf{2.9753} $\pm 0.1004$; 8.0 | Helped |
| Increase mixing from 2 points to 3 points (ablation) | Mixup · 2-class Gaussian, μ = 0.05 | NLL: 0.7487 $\pm 0.0356$ -> 0.7197 $\pm 0.0150$ | Helped |
| Increase mixing from 2 points to 5 points (ablation) | Mixup · 2-class Gaussian, μ = 0.05 and μ = 0.01 | NLL: 0.7487 $\pm 0.0356$ -> \textbf{0.6784} $\pm 0.0079$; 0.8247 $\pm 0.0400$ -> \textbf{0.7444} $\pm 0.0254$ | Helped |
| Increase mixing from 2 points to 5 points (ablation) | Mixup · 2-class Gaussian, μ = 0.25 | NLL: 0.1912 $\pm 0.0111$ -> 0.2729 $\pm 0.0299$ | Hurt |
| Increase label noise from 0% to 50% | ERM + temperature scaling · CIFAR-10, CIFAR-100, and SVHN | NLL: CIFAR-10: 2.0109 $\pm 0.1750$ -> 5.9533 $\pm 0.7209$; CIFAR-100: 5.4969 $\pm 0.8198$ -> 8.0224 $\pm 0.6025$; SVHN: 0.691 | Hurt |
| Increase label noise from 0% to 50% | Mixup · CIFAR-10, CIFAR-100, and SVHN | NLL: CIFAR-10: \textbf{0.8190} $\pm 0.0349$ -> \textbf{1.7729} $\pm 0.0773$; CIFAR-100: \textbf{2.4608} $\pm 0.0482$ -> \text | Hurt |
| Increase overlap (reduce μ) | ERM + temperature scaling · 2-class Gaussian | NLL: 0.2593 $\pm 0.0341$ -> 3.7331 $\pm 0.4986$ -> 4.2977 $\pm 0.5641$ | Hurt |
| Increase overlap (reduce μ) | 4-Mixup · 2-class Gaussian | NLL: 0.2729 $\pm 0.0299$ -> \textbf{0.6784} $\pm 0.0079$ -> \textbf{0.7444} $\pm 0.0254$ | Hurt |
Surprising findings
- On SVHN, ERM outperforms Mixup in test error yet still has much worse NLL, showing the gap is calibration rather than accuracy.
- d-Mixup with d>2 caused underconfidence on CIFAR and SVHN benchmarks, so only standard Mixup was reported there.
Limitations
- Theoretical results are asymptotic and rely on assumptions such as gamma-regularity and interpolation that are only empirically motivated.
- d-Mixup for large d is impractical; the paper conjectures standard Mixup suffices in practice without proof.
- Benchmark experiments use artificial label noise to induce overlap rather than natural overlaps.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 SVHN synthetic 2-class 300-dim Gaussian data
Benchmarks: CIFAR-10 CIFAR-100 SVHN synthetic Gaussian data
Metrics: NLL ECE ACE
Baselines: ERM + temperature scaling Mixup 2-Mixup 4-Mixup
Builds on: Mixup temperature scaling On Calibration of Modern Neural Networks
Closest papers by meaning
- On Temperature Scaling and Conformal Prediction of Deep Classifiers (ICML 2025 · similarity 0.90)
- Consistency-Guided Temperature Scaling Using Style and Content Information for Out-of-Domain Calibration (AAAI 2024 · similarity 0.87)
- T-CIL: Temperature Scaling using Adversarial Perturbation for Calibration in Class-Incremental Learning (CVPR 2025 · similarity 0.86)
- GETS: Ensemble Temperature Scaling for Calibration in Graph Neural Networks (ICLR 2025 · similarity 0.86)
- HoTS: Homophily-Aware Temperature Scaling for Graph Neural Network Calibration (NeurIPS 2026 · similarity 0.85)
- Beyond One-Hot Labels: Semantic Mixing for Model Calibration (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.