Research map / Architectures and efficiency / RNNs and normalization
Feature Clipping for Uncertainty Calibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes feature clipping (FC), a post-hoc calibration method that clips penultimate-layer feature values to a threshold tuned by NLL on validation data, raising entropy mainly for overconfident (high calibration error) samples.
Problem
Modern DNNs are overconfident and miscalibrated; existing calibration methods act on logits or outputs, while feature-level calibration is underexplored.
Approach
Clip feature values at a threshold c (tuned on validation NLL) so that high calibration error samples, which have larger feature variance and larger feature values, lose information and gain entropy while low calibration error samples are largely preserved; supported by a rectified-normal entropy analysis.
Key results
- On CIFAR-10 ResNet-50, feature clipping alone reduces ECE from 4.34 to 1.10 (vs. TS 1.39 base).
- Applied on top of Group Calibration, FC lowers ImageNet ResNet-50 ECE from 1.30 to 1.00 (best reported).
- FC keeps accuracy largely unchanged (e.g., CIFAR-10 ResNet-50 95.05 baseline vs 94.93 with FC), and gains on ViT-L-16 are small (ImageNet ECE 5.24 to 5.04).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Clip penultimate-layer features | ResNet-50 · CIFAR-10 | ECE (%): 4.34 -> 1.10 | Helped |
| Clip penultimate-layer features | ResNet-50 · CIFAR-100 | ECE (%): 17.52 -> 3.98 | Helped |
| Clip penultimate-layer features | ResNet-50 · ImageNet | ECE (%): 3.69 -> 1.74 | Helped |
| Add feature clipping to Group Calibration (ablation) | ResNet-110 · CIFAR-10 | ECE (%): 1.24 -> 1.78 | Hurt |
| Add feature clipping to Group Calibration (ablation) | DenseNet-121 · CIFAR-10 | ECE (%): 1.27 -> 2.51 | Hurt |
| Add feature clipping to Group Calibration (ablation) | ResNet-110 · CIFAR-100 | ECE (%): 2.71 -> 3.45 | Hurt |
| Add feature clipping (ablation) | ViT-L-16 · ImageNet | ECE (%): 5.24 -> 5.04 | Helped |
| Clip features at the penultimate layer rather than earlier layers (ablation) | ResNet-50 · CIFAR-10 | ECE (%): 4.42 -> 1.10; 14.42 -> 1.10 | Helped |
| Clip pre-antepenultimate-layer outputs rather than keep the baseline (ablation) | ResNet-50 · CIFAR-10 | accuracy (%): 95.05 -> 78.12 | Hurt |
| Vary the clipping threshold (ablation) | ResNet-50 · CIFAR-10 | ECE (%): ECE exceeding 25%; 1.10 | Mixed |
| Clip penultimate-layer features (ablation) | ResNet-50 and DenseNet-121 · CIFAR-10 and CIFAR-100 | accuracy (%) | No clear effect |
| Add feature clipping to train-time calibration methods (ablation) | CIFAR-10 models · CIFAR-10 | ECE (%): 1.53 -> 1.54; 2.97 -> 2.97 | Mixed |
Surprising findings
- Feature clipping gives little improvement on Vision Transformers, which the authors attribute to ViTs being less overconfident than CNNs (no ViT samples above 0.99 confidence).
- On several cases FC combined with Group Calibration worsens ECE, which the authors attribute to clipping reducing feature separability.
Limitations
- Limited improvement on transformer models such as ViT-L-16.
- Clipping threshold must be tuned on a validation set; the authors note faster and better threshold selection is needed.
- Theoretical analysis relies on assumed Gaussian pre-activations and an empirically observed sigma_HCE > sigma_LCE relation.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1K
Metrics: ECE Adaptive ECE accuracy NLL
Baselines: Temperature Scaling ETS PTS CTS Group Calibration Brier loss MMCE Label smoothing Focal loss FLSD-53 Dual Focal Loss
Builds on: temperature scaling label smoothing focal loss group calibration
Closest papers by meaning
- Uncertainty Weighted Gradients for Model Calibration (CVPR 2025 · similarity 0.86)
- A Benchmark Study on Calibration (ICLR 2024 · similarity 0.86)
- Calibration Bottleneck: Over-compressed Representations are Less Calibratable (ICML 2024 · similarity 0.86)
- C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion (ICLR 2024 · similarity 0.85)
- Decoupling Feature Extraction and Classification Layers for Calibrated Neural Networks (ICML 2024 · similarity 0.84)
- Reassessing How to Compare and Improve the Calibration of Machine Learning Models (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.