Research map / Architectures and efficiency / RNNs and normalization
Calibration Bottleneck: Over-compressed Representations are Less Calibratable
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that weight decay hurts model calibratability via over-compressed top layers, and proposes Progressively Layer-Peeled (PLP) training that gradually freezes top layers so the representation learns more calibratable features before temperature scaling.
Problem
Deep networks trained with weight decay and other regularizers become less amenable to post-hoc recalibration (low calibratability), creating a trade-off between calibratability and discriminability.
Approach
Identifies a U-shaped calibratability trend across hidden layers linked to information compression in top layers, and introduces the weak classifier hypothesis: a weakly trained head lets the representation learn more calibratable features; PLP progressively freezes top layers from top to bottom during training.
Key results
- PLP achieves the best average rank (1.94) across four datasets and two ResNet architectures, versus 4.44 for standard training, on post-TS calibrated metrics.
- On CIFAR-100 with ResNet-18, PLP reduces calibrated ECE to 0.96 from 2.61 for standard training, with accuracy 75.5 vs 73.8 for standard... (ResNet-18 CIFAR-100 PLP ECE 0.93 vs standard 2.61)
- Weight decay makes calibrated ECE worse (e.g., ResNet-18 CIFAR-100 calibrated ECE 4.60 vs 2.61 for standard training), while PLP maintains competitive accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PLP instead of standard training | ResNet-18 · SVHN | calibrated ECE: 0.44 -> 0.55 | Hurt |
| Use PLP instead of MIT | ResNet-18 · CIFAR-10 | calibrated ECE: 0.68 -> 0.58 | Helped |
| Use PLP instead of Mixup | ResNet-18 · CIFAR-100 | calibrated ECE: 1.18 -> 0.93 | Helped |
| Use PLP instead of Distillation | ResNet-18 · Tiny-ImageNet | calibrated ECE: 1.14 -> 0.96 | Helped |
| Use PLP instead of standard training | ResNet-50 · SVHN | calibrated ECE: 0.64 -> 0.63 | No clear effect |
| Use PLP instead of MIT | ResNet-50 · CIFAR-10 | calibrated ECE: 0.66 -> 0.48 | Helped |
| Use PLP instead of Focal Loss | ResNet-50 · CIFAR-100 | calibrated ECE: 1.13 -> 1.09 | Helped |
| Use PLP instead of standard training | ResNet-50 · Tiny-ImageNet | calibrated ECE: 1.66 -> 1.10 | Helped |
| Increase weight decay strength | ResNet-18 · CIFAR-100 | calibrated ECE | Hurt |
| Freeze top layers earlier during PLP training (increase γ) (ablation) | PLP-trained ResNet-18 · CIFAR-100 | calibrated ECE | Helped |
| Freeze top layers earlier during training (ablation) | ResNet-18 · CIFAR-100 | calibrated ECE | Helped |
| Freeze top layers earlier during training (ablation) | ResNet-18 · CIFAR-100 | accuracy | Hurt |
Surprising findings
- Weight decay, the standard regularizer recommended by Guo et al. (2017) for calibration, reduces calibratability: raw ECE improves with regularization but post-TS calibrated ECE worsens.
- Methods like label smoothing, distillation and MMCE, designed for train-time calibration, often perform worse than standard training after post-hoc calibration.
Limitations
- Experiments limited to small image classification datasets and architectures; larger datasets and pretraining not studied.
- Top-layer early stopping is sensitive to the choice of cut point and freezing epoch; PLP relies on heuristic phase partitioning controlled by gamma.
- Distribution shift calibration not evaluated.
Experimental setup
Datasets: SVHN CIFAR-10 CIFAR-100 Tiny-ImageNet
Benchmarks: SVHN CIFAR-10 CIFAR-100 Tiny-ImageNet
Metrics: calibrated ECE calibrated AECE calibrated NLL accuracy
Baselines: Standard training Weight Decay Brier loss MMCE Distillation Label Smoothing Focal Loss Mixup MIT
Builds on: temperature scaling layer-peeled model information bottleneck
Closest papers by meaning
- A Benchmark Study on Calibration (ICLR 2024 · similarity 0.89)
- Uncertainty Weighted Gradients for Model Calibration (CVPR 2025 · similarity 0.88)
- Reassessing How to Compare and Improve the Calibration of Machine Learning Models (ICLR 2025 · similarity 0.87)
- Feature Clipping for Uncertainty Calibration (AAAI 2025 · similarity 0.86)
- Decoupling Feature Extraction and Classification Layers for Calibrated Neural Networks (ICML 2024 · similarity 0.86)
- Tilt and Average : Geometric Adjustment of the Last Layer for Recalibration (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.