Research map / Label-efficient and robust learning / Recognition and long-tail

On Temperature Scaling and Conformal Prediction of Deep Classifiers

ICML 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Confidence calibration objectives · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Empirically and theoretically studies how temperature scaling affects adaptive conformal prediction (APS, RAPS) set size and class-conditional coverage, showing a non-monotonic trend in set size and a trade-off controllable via temperature.

Problem

Calibration (temperature scaling) and conformal prediction are both used to quantify uncertainty of DNN classifiers, but their interplay has not been studied; in particular, how TS affects set size and conditional coverage of CP methods is unknown.

Approach

Shows that TS calibration increases APS/RAPS set sizes while improving class-conditional coverage, explores TS over a wide temperature range beyond calibration, and provides a theory (threshold monotonicity plus a gap-function analysis) explaining the non-monotonic set size trend, leading to practitioner guidelines using a separate CP temperature.

Key results

• After TS calibration, APS set size on CIFAR-100 ResNet50 rises from 5.31 to 9.14 at alpha=0.1, while LAC changes negligibly (1.62 to 1.57).
• TS calibration improves APS/RAPS top-5% class coverage gap (e.g., ImageNet ViT-B/16 APS 14.2% to 12.2%), while marginal coverage is maintained within 0.1%.
• Sweeping temperature from 0.5 to 5 shows a non-monotonic AvgSize peak for APS and RAPS, with the threshold decreasing monotonically; theory explains the peak location and its shift with the number of classes C.

Measured effects reported in the paper

Surprising findings

• Temperature scaling, designed for calibration, increases the average prediction set size of APS and RAPS even though it improves class-conditional coverage.
• The effect of temperature on APS/RAPS set size is non-monotonic, with set size peaking at a critical temperature before declining.

Limitations

• Theory covers only the set size of APS/RAPS; the effect on conditional coverage is left for future work.
• Theoretical analysis relies on an assumption that the threshold-defining quantile sample is shared with and dominated by a large logit gap, approximated rather than proven.
• Experiments are limited to image classifiers on CIFAR-10/100 and ImageNet.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 ImageNet

Benchmarks: CIFAR-10 CIFAR-100 ImageNet

Metrics: AvgSize MarCovGap TopCovGap AvgCovGap

Baselines: LAC APS RAPS Mondrian CP

Builds on: Temperature Scaling Adaptive Prediction Sets Regularized Adaptive Prediction Sets Least Ambiguous set-valued Classifier

Closest papers by meaning

• On the Limitations of Temperature Scaling for Distributions with Overlaps (ICLR 2024 · similarity 0.90)
• Class Adaptive Conformal Training (NeurIPS 2026 · similarity 0.88)
• Self-Calibrating Conformal Prediction (NeurIPS 2024 · similarity 0.88)
• Direct Prediction Set Minimization via Bilevel Conformal Classifier Training (ICML 2025 · similarity 0.88)
• Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration (NeurIPS 2024 · similarity 0.87)
• Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence (ICLR 2025 · similarity 0.87)