Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Pointwise Information Measures as Confidence Estimators in Deep Neural Networks: A Comparative Study
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares three post-hoc pointwise information measures (PMI, PVI, PSI) as confidence scores for failure prediction and calibration of image classifiers, finding PVI best for failure prediction and on par with temperature-scaled softmax for calibration.
Problem
Confidence estimation for deep neural network predictions is often poorly calibrated, and existing calibration methods can harm failure prediction; post-hoc confidence measures that avoid architecture or training changes are needed.
Approach
Use pointwise information measures (pointwise mutual information, pointwise V-information, pointwise sliced mutual information) between penultimate features or inputs and predicted labels as post-hoc confidence scores, with theoretical analysis of invariance, convergence rates and margin sensitivity, and a comparison against softmax-based baselines.
Key results
- PVI achieves the best failure prediction on most settings, e.g., ResNet50 on CIFAR-10 AUROCf 95.14 vs 93.59 for MSP and AUPRf,error 60.99 vs 43.42.
- On ResNet101 CIFAR-100, PVI reaches AUROCf 89.02 vs 85.54 for MSP and E-AURC 30.44 vs 39.17.
- For calibration, PVI matches temperature-scaled MSP on most datasets (e.g., ResNet50 CIFAR-10 SCE 0.17 vs 0.19), with differences typically small.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PVI instead of PMI | MLP confidence estimation · MNIST | AUROCf × 102: 97.13 ± 0.14 -> 96.94 ± 0.22 | Hurt |
| Use PVI instead of MSP | CNN confidence estimation · F-MNIST | AUROCf × 102: 92.53 ± 0.17 -> 92.98 ± 0.52 | Helped |
| Use PVI instead of MSP | VGG16 confidence estimation · STL-10 | AUROCf × 102: 92.04 ± 0.25 -> 92.79 ± 0.62 | Helped |
| Use PVI instead of logits margin | ResNet50 confidence estimation · CIFAR-10 | AUROCf × 102: 93.80 ± 0.23 -> 95.14 ± 0.30 | Helped |
| Use PVI instead of softmax margin | ResNet101 confidence estimation · CIFAR-100 | AUROCf × 102: 85.92 ± 0.29 -> 89.02 ± 0.38 | Helped |
| Use PVI instead of softmax margin | InceptionV3 confidence estimation · Stanford Dogs | AUROCf × 102: 83.63 ± 0.37 -> 85.73 ± 0.73 | Helped |
| Use PVI instead of MSP | DenseNet121 confidence estimation · TinyImageNet | AUROCf × 102: 87.01 ± 0.34 -> 89.66 ± 0.27 | Helped |
| Use PVI instead of PMI | MLP confidence estimation · MNIST | SCE: 0.05 ± 0.01 -> 0.05 ± 0.01 | No clear effect |
| Use PVI instead of MSP | CNN confidence estimation · F-MNIST | SCE: 0.15 ± 0.02 -> 0.14 ± 0.02 | Helped |
| Use PVI instead of PSI | VGG16 confidence estimation · STL-10 | SCE: 0.22 ± 0.05 -> 0.20 ± 0.03 | Helped |
| Use PVI instead of PMI | ResNet50 confidence estimation · CIFAR-10 | SCE: 0.15 ± 0.01 -> 0.17 ± 0.00 | Hurt |
| Use PVI instead of PSI | ResNet101 confidence estimation · CIFAR-100 | SCE: 0.07 ± 0.01 -> 0.08 ± 0.00 | Hurt |
Surprising findings
- PSI, despite being most sensitive to sample-wise margin, performs poorly in failure prediction, suggesting margin sensitivity alone is not a sufficient criterion for good confidence measures.
- Post-hoc invariance to general homeomorphisms (PMI) may be counterproductive for confidence estimation, as the degree of non-linearity can indicate network confidence.
Limitations
- PI measures require training additional models to estimate the probability distributions (e.g., two networks for PVI), adding cost.
- PMI is omitted on datasets with many classes (CIFAR-100, Stanford Dogs, TinyImageNet) due to computational cost.
- Margin correlation and confidence-as-reliability interpretations may disagree; PSI's margin sensitivity does not translate to better failure prediction.
Experimental setup
Datasets: MNIST Fashion-MNIST STL-10 CIFAR-10 CIFAR-100 Stanford Dogs TinyImageNet
Benchmarks: MNIST Fashion-MNIST STL-10 CIFAR-10 CIFAR-100 Stanford Dogs TinyImageNet
Metrics: AUROC AUPR FPR95 E-AURC SCE CC-SCE Ada-SCE RMS-SCE
Baselines: MSP softmax margin max logit logits margin negative entropy negative Gini index PMI PSI
Builds on: V-information sliced mutual information pointwise V-information (Ethayarajh et al., 2022) temperature scaling
Closest papers by meaning
- Learning for Single-Shot Confidence Calibration in Deep Neural Networks Through Stochastic Inferences (CVPR 2019 · similarity 0.86)
- SURE: SUrvey REcipes for building reliable and robust deep networks (CVPR 2024 · similarity 0.85)
- High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach (ICML 2018 · similarity 0.85)
- Confidence-Aware Learning for Deep Neural Networks (ICML 2020 · similarity 0.85)
- Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence (ICLR 2025 · similarity 0.83)
- InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.