Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
A Rate-Distortion View of Uncertainty Quantification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Distance Aware Bottleneck (DAB), which learns a rate-distortion codebook of distributional centroids over training-set encoders so that KL distance to the codebook gives a single-pass, distance-aware uncertainty score for OOD and misclassification detection.
Problem
Deep neural networks lack distance awareness, so they cannot reliably signal when an input is far from the training data; existing deterministic uncertainty methods often rely on constraints that hurt calibration or do not scale to large pretrained models.
Approach
Frames uncertainty as a rate-distortion problem over the distribution of encoders: a codebook of k Gaussian code distributions (learned with alternating minimization, as in the variational information bottleneck extended by RDFC) is trained, and the expected KL distance from an input's encoder to the codes is the uncertainty score.
Key results
- On CIFAR-10 vs SVHN, DAB reaches AUROC 0.986 and AUPRC 0.994, above the 5-model deep ensemble (0.97 AUROC) and all single-pass DUMs.
- On CIFAR-10 misclassification prediction, DAB achieves calibration AUROC 0.930, versus 0.951 for the deep ensemble and 0.897 for SNGP.
- On ImageNet-1K, DAB with a frozen pre-trained ResNet-50 reaches calibration AUROC 0.866 with about 13M trainable parameters, versus 0.861 for a 5-model ensemble with about 118M parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DAB's distributional codebook distance for uncertainty estimation instead of the strongest listed baseline | Wide ResNet 28-10 · CIFAR-10 vs SVHN | AUROC: 0.981 ± 0.002 -> 0.986 ± 0.004 | Helped |
| Use DAB's distributional codebook distance for uncertainty estimation instead of the strongest listed baseline | Wide ResNet 28-10 · CIFAR-10 vs SVHN | AUPRC: 0.987 ± 0.001 -> 0.994 ± 0.002 | Helped |
| Use DAB's distributional codebook distance for uncertainty estimation instead of the strongest listed baseline | Wide ResNet 28-10 · CIFAR-10 vs CIFAR-100 | AUROC: 0.916 ± 0.001 -> 0.922 ± 0.002 | Helped |
| Use DAB's distributional codebook distance for uncertainty estimation instead of the strongest listed baseline | Wide ResNet 28-10 · CIFAR-10 vs CIFAR-100 | AUPRC: 0.907 ± 0.002 -> 0.915 ± 0.002 | Helped |
| Use DAB's statistical distance from centroids for misclassification prediction | Wide ResNet 28-10 · CIFAR-10 misclassification prediction | Calibration AUROC: 0.951 ± 0.001 -> 0.930 ± 0.003 | Hurt |
| Use DAB with fine-tuned ResNet-50 for misclassification prediction | ResNet-50 · ImageNet-1K misclassification prediction | Calibration AUROC: 0.861 ± 0.0004 -> 0.868 ± 0.0008 | Helped |
| Use DAB with fine-tuned ResNet-50 for OOD detection | ResNet-50 · ImageNet-1K vs ImageNet-O | OOD AUROC: 0.642 ± 0.001 -> 0.743 ± 0.004 | Helped |
| Use DAB with pre-trained, frozen ResNet-50 features rather than fine-tuning the backbone | DAB with ResNet-50 · ImageNet-1K misclassification prediction | Calibration AUROC: 0.868 ± 0.0008 -> 0.866 ± 0.0003 | Hurt |
| Use DAB with pre-trained, frozen ResNet-50 features rather than fine-tuning the backbone | DAB with ResNet-50 · ImageNet-1K vs ImageNet-O | OOD AUROC: 0.743 ± 0.004 -> 0.732 ± 0.004 | Hurt |
Surprising findings
- Training DAB with a frozen pre-trained ResNet-50 backbone costs little in OOD or calibration performance relative to fine-tuning, and the ImageNet-O AUROC only drops from 0.743 to 0.732.
Limitations
- Accuracy still lags deep ensembles, as with other single-pass DUMs.
- Experiments are mainly on image classification; applications to language and other modalities are left to future work.
- Uses only KL as the distortion measure and does not use outlier exposure during training.
Experimental setup
Datasets: CIFAR-10 SVHN CIFAR-100 ImageNet-1K ImageNet-O
Benchmarks: CIFAR-10 vs SVHN OOD CIFAR-10 vs CIFAR-100 OOD CIFAR-10 misclassification prediction ImageNet-1K misclassification prediction ImageNet vs ImageNet-O OOD
Metrics: AUROC AUPRC Calibration AUROC accuracy trainable parameters
Baselines: Deep Ensemble of 5 Deterministic DDU DUQ DUE SNGP vanilla VIB
Builds on: Variational Information Bottleneck Rate Distortion Finite Cardinality Blahut-Arimoto algorithm Deterministic Uncertainty Quantification (DUQ) Spectral-normalized Neural Gaussian Process (SNGP) Deep Ensembles
Closest papers by meaning
- Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (NeurIPS 2020 · similarity 0.87)
- Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression (ICLR 2024 · similarity 0.86)
- Uncertainty Quantification with the Empirical Neural Tangent Kernel (NeurIPS 2025 · similarity 0.86)
- Cauchy-Schwarz Divergence Information Bottleneck for Regression (ICLR 2024 · similarity 0.85)
- Structured IB: Improving Information Bottleneck with Structured Feature Learning (AAAI 2025 · similarity 0.85)
- Combining Statistical Depth and Fermat Distance for Uncertainty Quantification (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.