Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
General Uncertainty Estimation with Delta Variances
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Delta Variances, a family of gradient-based epistemic uncertainty estimators for quantities of interest computed from neural networks, unifying Bayesian, frequentist, adversarial and OOD views, and validates them on GraphCast weather forecasting.
Problem
Estimating epistemic uncertainty efficiently for large neural networks and for downstream quantities of interest computed from learned models, where ensembles and MC dropout incur multiplicative inference cost.
Approach
Approximate the epistemic variance of any differentiable quantity of interest u_theta(z) as a quadratic form of its parameter gradient with a fixed covariance Sigma (e.g. inverse empirical Fisher or sandwich/Hessian forms), requiring only one gradient computation and no change to architecture or training; the family is derived from Delta Method, leave-one-out/jackknife, bootstrap, adversarial and
Key results
- Delta Variances with diagonal Fisher approximation give results competitive with ensembles at the cost of a single gradient pass on the GraphCast-based weather benchmark.
- Ensembles achieve the highest quality on average, while Delta Variances have the lowest computational overhead.
- Fine-tuning per-block scale factors of Sigma improves the Delta Variance results over the regular Fisher information.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Delta Variances instead of ensemble methods | GraphCast weather forecasting uncertainty estimation · GraphCast weather forecasting | prediction quality | No clear effect |
| Use ensemble methods instead of Delta Variances | GraphCast weather forecasting uncertainty estimation · GraphCast weather forecasting | prediction quality | Helped |
| Replace the Fisher-based covariance with a learned or fine-tuned covariance | Delta Variances · GraphCast weather forecasting | uncertainty prediction quality | Helped |
| Rescale covariance entries by layer using fine-tuning (ablation) | Delta Variances · GraphCast weather forecasting | log-likelihood | Helped |
Surprising findings
- Delta Variance of a quantity of interest is harder to estimate than the variance of the learned function itself, in the illustrative survival example.
Limitations
- Ensembles still achieve higher quality than Delta Variances on average
- GraphCast was retrained at 4-degree grid with reduced layers and no fine-tuning curriculum, departing from the original system
- Theoretical guarantees rely on assumptions (local convergence, bounded derivatives) that may not hold for neural networks
Experimental setup
Datasets: ERA5-style weather reanalysis (1979-2021, as used by GraphCast)
Benchmarks: weather forecasting quantities of interest (252 QoIs, 2018-2021 holdout)
Metrics: log-likelihood correlation between absolute error and predicted epistemic s AUC of error removal
Baselines: Bootstrapped Ensemble MC-Dropout
Builds on: GraphCast Laplace approximation infinitesimal jackknife SWAG Monte-Carlo dropout
Closest papers by meaning
- One Step Closer to Unbiased Aleatoric Uncertainty Estimation (AAAI 2024 · similarity 0.87)
- Epistemic Uncertainty Quantification For Pre-Trained Neural Networks (CVPR 2024 · similarity 0.87)
- Adaptive Sampling to Reduce Epistemic Uncertainty Using Prediction Interval-Generation Neural Networks (AAAI 2025 · similarity 0.87)
- Uncertainty Quantification with the Empirical Neural Tangent Kernel (NeurIPS 2025 · similarity 0.86)
- Uncertainty Quantification via Stable Distribution Propagation (ICLR 2024 · similarity 0.85)
- Estimating Epistemic and Aleatoric Uncertainty with a Single Model (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.