Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Epistemic Uncertainty Quantification For Pre-Trained Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a gradient-based epistemic uncertainty score for pre-trained non-Bayesian classifiers (REGrad with class-specific L2 weighting, layer-selective gradients, and input-perturbation gradient smoothing), evaluated on OOD detection, calibration, and active learning on small image benchmarks.
Problem
Quantifying epistemic uncertainty for any pre-trained non-Bayesian neural network without original training data, model modifications, or architecture constraints; existing gradient- and perturbation-based methods lack theoretical grounding and often underperform.
Approach
Connects perturbation and gradient UQ to BNN mutual information theoretically, then improves gradient-based UQ with an L2 root-normalized class-probability weighting of per-class gradients, exponentially increasing weights for deeper layers, and averaging gradients over Gaussian-perturbed inputs.
Key results
- REGrad* (REGrad + layer-selective + perturbation) reaches average OOD AUROC 94.56 on MNIST-based pairs and 85.91 on SVHN/C100-based pairs, outperforming ExGrad (91.86 and 81.02).
- In active learning, REGrad* achieves 48.83 average ACC across four datasets, versus 47.11 for ExGrad and 44.06 for LA.
- Calibration rAULC averages 0.904 for REGrad*, versus 0.897 for ExGrad and 0.874 for LA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use REGrad* instead of the strongest reported baseline | pre-trained models · OOD detection (first Table 1 block, Avg) | AUROC: 92.85 -> 94.56 | Helped |
| use REGrad* instead of the strongest reported baseline | pre-trained models · OOD detection (second Table 1 block, Avg) | AUROC: 81.02 -> 85.91 | Helped |
| use REGrad* instead of Entropy | pre-trained models · uncertainty calibration (Avg) | rAULC: 0.902 -> 0.904 | Helped |
| use REGrad* instead of Inserted Dropout | active learning · MNIST, C10, SVHN, and C100 (Avg across 10 acquisition cycles) | ACC: 47.69 -> 48.83 | Helped |
| use REGrad* instead of LA | active learning · MNIST, C10, SVHN, and C100 (Avg across 10 acquisition cycles) | NLL: 2.592 -> 2.547 | Helped |
| replace ExGrad with REGrad class-specific root-normalized probability weighting (ablation) | gradient-based UQ · MNIST → FMNIST, C10 → SVHN, SVHN → C10, and C100 → SVHN | AUROC: 98.11 -> 98.42; 88.87 -> 89.64; 88.57 -> 90.22; 79.99 -> 81.43 | Helped |
| add layer-selective gradients to REGrad (ablation) | REGrad · MNIST → FMNIST, C10 → SVHN, SVHN → C10, and C100 → SVHN | AUROC: 98.42 -> 98.51; 89.64 -> 90.33; 90.22 -> 90.72; 81.43 -> 87.32 | Helped |
| add input perturbations and aggregate gradients (ablation) | REGrad + layer-selective gradients · MNIST → FMNIST, C10 → SVHN, SVHN → C10, and C100 → SVHN | AUROC: 98.51 -> 98.78; 90.33 -> 92.32; 90.72 -> 91.11; 87.32 -> 88.06 | Helped |
| set layer-weighting coefficient λ → ∞, emphasizing only the last layer (ablation) | REGrad + layer-selective gradients · SVHN → C10 | AUROC: 90.72 -> 88.21 | Hurt |
| vary λ within 0.25–0.35 (ablation) | REGrad + layer-selective gradients · SVHN → C10 | AUROC: 90.65; 90.72; 90.80 | No clear effect |
| vary perturbation scale σ within .015–.025 (ablation) | REGrad + layer-selective gradients · SVHN → C10 | AUROC: 91.05; 91.11; 90.98 | No clear effect |
Surprising findings
- Gradient-based GradNorm performs far below chance-level or near random on several OOD pairs (average AUROC 25.12 on the first table), and UNGrad is unstable across pairs.
- Improvements over the strongest baselines are modest in calibration (about 0.007 rAULC over ExGrad), and some baselines like Entropy are close to the proposed method.
Limitations
- Theoretical guarantees rely on strong assumptions (infinite training data, Bernstein-von Mises regime) that rarely hold exactly in practice.
- Evaluation limited to small image classification benchmarks with CNNs and ResNets (vision transformers only in appendix).
- Hyperparameters (lambda, sigma) must be tuned and performance depends on their ranges.
Experimental setup
Datasets: MNIST SVHN CIFAR-10 CIFAR-100 Omniglot Fashion-MNIST LSUN
Benchmarks: MNIST vs Omniglot OOD MNIST vs FMNIST OOD CIFAR-10 vs SVHN OOD CIFAR-10 vs LSUN OOD SVHN vs CIFAR-10 OOD SVHN vs CIFAR-100 OOD CIFAR-100 vs SVHN OOD CIFAR-100 vs LSUN OOD uncertainty calibration rAULC on MNIST, C10, SVHN, C100 active learning ACC/NLL on MNIST, C10, SVHN, C100
Metrics: AUROC AUPR rAULC ACC NLL
Baselines: NEGrad UNGrad GradNorm ExGrad Perturb x Perturb θ MC-AA Inserted Dropout Entropy LA
Builds on: ExGrad Laplace approximation Bernstein-von Mises theorem SmoothGrad
Closest papers by meaning
- Uncertainty Quantification with the Empirical Neural Tangent Kernel (NeurIPS 2025 · similarity 0.91)
- Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage? (NeurIPS 2024 · similarity 0.89)
- Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods? (ICML 2024 · similarity 0.88)
- Accurate and Scalable Estimation of Epistemic Uncertainty for Graph Neural Networks (ICLR 2024 · similarity 0.87)
- General Uncertainty Estimation with Delta Variances (AAAI 2025 · similarity 0.87)
- Evaluating Epistemic Uncertainty: Beyond OOD Detection and Active Learning (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.