Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks 19 uncertainty quantification methods and their aleatoric/epistemic decomposition formulas on ImageNet-1k and CIFAR-10, finding that existing decompositions yield highly correlated, non-disentangled estimators and that specialized tasks remain harder than saturated predictive tasks.
Problem
Existing uncertainty quantification work evaluates methods on single uncertainty components or small-scale experiments, so it is unclear whether decomposition formulas yield truly disentangled aleatoric and epistemic estimators and which methods work best for specific uncertainty tasks in practice.
Approach
Reimplements a broad set of distributional and deterministic uncertainty methods under a common ResNet-50 / ImageNet pipeline and evaluates them across thirteen practical tasks (OOD detection, aleatoric rank correlation with human label entropy, correctness prediction, abstention, calibration, robustness), quantifying disentanglement via rank correlation between decomposed components.
Key results
- All twelve distributional methods on ImageNet show high rank correlation (>=0.78) between IT aleatoric and epistemic estimates; eleven of twelve on CIFAR-10 show >=0.93.
- Mahalanobis is the best OOD detector on ImageNet-C severity-two corruptions, while DDU is the worst (AUROC 0.675).
- On correctness prediction, EDL and PostNet are considerably better than other methods, but lose their advantage at matched accuracy; most abstention methods (except Mahalanobis) exceed AUAC 0.91.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Mahalanobis method for severity-two OOD detection | ImageNet-1k OOD detection · ImageNet-C severity 2 | AUROC | Helped |
| Use DDU for OOD detection | ImageNet-1k OOD detection · ImageNet-C severity 2 | AUROC: AUROC $= 0.675$ | Hurt |
| Use IT decomposition to estimate aleatoric and epistemic uncertainty | Distributional methods · ImageNet-1k | rank correlation between aleatoric and epistemic estimates: rank corr. $in [0.78, 0.99]$ | Hurt |
| Combine Mahalanobis with the CE baseline | ImageNet-1k uncertainty estimator pair · ImageNet-1k | rank correlation between estimator pair: 0.15 ± 0.01 | Helped |
| Use EDL or PostNet for correctness prediction | ImageNet-1k correctness prediction · ImageNet-1k validation | AUROC | Helped |
| Use EDL or PostNet for abstained prediction | ImageNet-1k abstained prediction · ImageNet-1k validation | AUAC | Hurt |
| Use Laplace or temperature scaling for calibration | ImageNet-1k calibration · ImageNet-1k | expected calibration error | Helped |
| Use the IT decomposition estimators instead of the best-performing estimators | Shallow Ensemble · ImageNet-1k OOD detection and aleatoric uncertainty | performance | Hurt |
| Use uncertainty estimates to rank correctness under ImageNet-C perturbations | ImageNet uncertainty estimators · ImageNet-C | correctness prediction AUROC | No clear effect |
| Use uncertainty estimates to rank correctness under CIFAR-10C perturbations | CIFAR-10 uncertainty estimators · CIFAR-10C | correctness prediction AUROC | Hurt |
Surprising findings
- No tested decomposition formula yields disentangled aleatoric and epistemic estimators on any tested second-order distribution.
- Nine of thirteen tasks show substantially different method rankings between CIFAR-10 and ImageNet (rank corr. <= 0.5).
Limitations
- Only two classification datasets are used because aleatoric ground truths require multiple annotations per input.
- Models are trained to convergence on large-scale data; the small-data regime where epistemic uncertainty is far from converged is not covered in the main study.
Experimental setup
Datasets: ImageNet-1k CIFAR-10 ImageNet-C ImageNet-ReaL CIFAR-10H CIFAR-10C
Benchmarks: ImageNet-1k CIFAR-10 ImageNet-C CIFAR-10C ImageNet-ReaL CIFAR-10H
Metrics: AUROC rank correlation AUAC ECE Pearson correlation rAULC E-AURC
Baselines: Deep Ensemble MC Dropout SNGP Laplace EDL PostNet Mahalanobis Temperature Scaling
Builds on: Deep Ensemble Laplace approximation Evidential Deep Learning SNGP Mahalanobis
Closest papers by meaning
- From Risk to Uncertainty: Generating Predictive Uncertainty Measures via Bayesian Estimation (ICLR 2025 · similarity 0.89)
- Evaluating Epistemic Uncertainty: Beyond OOD Detection and Active Learning (NeurIPS 2026 · similarity 0.88)
- Predictive Uncertainty Quantification for Bird's Eye View Segmentation: A Benchmark and Novel Loss Function (ICLR 2025 · similarity 0.87)
- ValUES: A Framework for Systematic Validation of Uncertainty Estimation in Semantic Segmentation (ICLR 2024 · similarity 0.86)
- Hyperdimensional Uncertainty Quantification for Multimodal Uncertainty Fusion in Autonomous Vehicles Perception (CVPR 2025 · similarity 0.86)
- UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Model (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.