Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Streamlining Prediction in Bayesian Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes streamlined BDL prediction via layer-wise local linearisation of activations and local Gaussian approximations at linear layers, giving analytic posterior predictive in a single forward pass without sampling.
Problem
Monte Carlo integration remains the standard for computing posterior predictive distributions in Bayesian deep learning, which is costly and cumbersome for downstream inference tasks such as outlier detection and sensitivity analysis.
Approach
Propagate Gaussian posterior uncertainty through the network layer by layer: local Gaussian approximation at linear layers (assuming activation-parameter independence) and first-order Taylor linearisation of activations, yielding an analytic predictive distribution in one forward pass; extends to attention by treating queries and keys deterministically and propagating values.
Key results
- On UCI regression with MFVI, the method wins NLPD on 11/11 datasets versus 3/11 for sampling; with LA it wins 8/11 versus GLM's 7/11.
- On MNIST-like classification, the method matches sampling accuracy while achieving lower ECE on most datasets (e.g., 0.006 vs 0.010 on FMNIST for MFVI).
- On GPT-2 language tasks, the method underperforms GLM on accuracy (e.g., MRPC 0.416 vs 0.720 for LA), so gains are not uniform.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use local Gaussian approximation and local linearisation instead of posterior sampling | MFVI on UCI regression · UCI regression datasets | NLPD: Servo: 1.287 -> 1.136; LD: 1.346 -> 1.369; FF: 2.160 -> 2.333 | Mixed |
| Use local Gaussian approximation and local linearisation instead of posterior sampling | LA on UCI regression · UCI regression datasets | NLPD: Servo: 3.795 -> 1.443; CAC: 1.848 -> 2.662 | Mixed |
| Use local Gaussian approximation and local linearisation instead of posterior sampling | LA and MFVI classification · MNIST-like datasets | NLPD: LA: FMNIST 0.369 -> 0.331; MFVI: FMNIST 0.311 -> 0.313 | Mixed |
| Use local Gaussian approximation and local linearisation instead of posterior sampling | LA on ViT attention layers · CIFAR-10 | NLPD: 0.099 -> 0.087 | Helped |
| Use local Gaussian approximation and local linearisation instead of posterior sampling | MFVI on ViT attention layers · CIFAR-100 | NLPD: 0.568 -> 0.451 | Helped |
| Use local Gaussian approximation and local linearisation instead of posterior sampling | MFVI on ViT attention layers · DTD | NLPD: 0.967 -> 0.909 | Helped |
| Use local Gaussian approximation and local linearisation instead of posterior sampling | MFVI on ViT attention layers · RESISC | NLPD: 0.314 -> 0.272 | Helped |
| Use local Gaussian approximation and local linearisation instead of posterior sampling | LA on ViT attention layers · ImageNet-R | NLPD: 1.262 -> 1.192 | Helped |
| Use local Gaussian approximation and local linearisation instead of GLM | LA on GPT-2 · BOOL-Q | accuracy: 0.678 -> 0.622 | Hurt |
| Use local Gaussian approximation and local linearisation instead of GLM | LA on GPT-2 · MRPC | accuracy: 0.720 -> 0.416 | Hurt |
| Replace moment-matching with local linearisation (ablation) | Bayesian MLP classification · MNIST | accuracy: 0.949 -> 0.976 | Helped |
| Replace moment-matching with local linearisation (ablation) | Bayesian MLP classification · OrganCMNIST | ECE: 0.190 -> 0.099 | Hurt |
Surprising findings
- On GPT-2 language tasks, the proposed method's accuracy is substantially below the GLM baseline on BoolQ, WiC and MRPC, and near chance on MRPC for LA.
Limitations
- Local linearisation error depends on activation function and input distribution scale.
- Independence assumption between activations and parameters ignores correlations, especially in residual blocks.
- Requires a validation set and grid search to fit the predictive variance scaling factor.
Experimental setup
Datasets: UCI regression datasets MNIST FMNIST OrganCMNIST OrganSMNIST CIFAR-10 CIFAR-100 DTD RESISC ImageNet-R BoolQ WiC
Benchmarks: UCI regression MNIST FMNIST OrganCMNIST OrganSMNIST CIFAR-10 CIFAR-100 DTD RESISC ImageNet-R BoolQ WiC
Metrics: NLPD RMSE ACC ECE predictive entropy
Baselines: Monte Carlo sampling GLM (linearised Laplace) moment-matching
Builds on: Laplace approximation linearised Laplace approximation IVON Kronecker-factored approximate curvature
Closest papers by meaning
- Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI (ICML 2024 · similarity 0.87)
- Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shift (NeurIPS 2023 · similarity 0.87)
- Scalable Bayesian Learning with posteriors (ICLR 2025 · similarity 0.86)
- Variational Bayesian Pseudo-Coreset (ICLR 2025 · similarity 0.85)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.85)
- Bayesian Low-Rank Learning (Bella): A Practical Approach to Bayesian Neural Networks (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.