Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Do Bayesian Neural Networks Actually Behave Like Bayesian Models?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically tests whether popular approximate BNN inference methods (MFVI, Laplace, SWAG, SGLD, HMC) obey Bayesian properties such as functional variability, sequential coherence, and martingale predictive coherence on synthetic regression, CIFAR-10, and IMDB.
Problem
Approximate inference for Bayesian neural networks is known to be inexact, but it is unclear whether BNNs behave like true Bayesian models in sequential updating, information retention, and predictive coherence, which matter for active, continual, and transfer learning.
Approach
Designs diagnostics beyond static accuracy: function-space posterior variability, sequential-vs-batch update consistency, and a martingale-posterior predictive coherence test (label imputation and retraining) to check whether BNN updates respect Bayesian belief updating.
Key results
- Iterated MFVI on CIFAR-10 and IMDB performs significantly and consistently worse than single-step MFVI across accuracy, log-likelihood, and ECE.
- In the martingale posterior test on CIFAR-10, the BMA metrics of imputed-label retraining deviate from the initial posterior beyond bootstrapped confidence intervals for both VI and SGLD, violating predictive coherence.
- In the intermediary Laplace experiment, after only 25 new points the posterior predictive mean turns negative, indicating new observations are weighted about twice as much as 50 pretraining points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Update MFVI sequentially on two data splits instead of fitting one split jointly | MFVI · CIFAR-10 | accuracy | Hurt |
| Update MFVI sequentially on two data splits instead of fitting one split jointly | MFVI · CIFAR-10 | log-likelihood | Hurt |
| Update MFVI sequentially on two data splits instead of fitting one split jointly | MFVI · CIFAR-10 | ECE | Hurt |
| Update MFVI sequentially on two data splits instead of fitting one split jointly | MFVI · IMDB | accuracy | Hurt |
| Update MFVI sequentially on two data splits instead of fitting one split jointly | MFVI · IMDB | log-likelihood | Hurt |
| Update MFVI sequentially on two data splits instead of fitting one split jointly | MFVI · IMDB | ECE | Hurt |
| Use an MFVI initialization from the first split for SGD training on the second split | SGD · CIFAR-10 and IMDB | predictive performance | Helped |
| Average retrained MFVI posteriors after pseudo-label imputation | MFVI · CIFAR-10 subset | accuracy | Helped |
| Average retrained SGLD posteriors after pseudo-label imputation | SGLD · CIFAR-10 subset | accuracy | Helped |
| Use Laplace posterior approximation as the prior for subsequent inference on shifted data | Laplace followed by HMC · Synthetic regression | retention of earlier observations | Hurt |
| Use MFVI posterior approximation as the prior for subsequent inference on shifted data | MFVI · Synthetic regression | retention of earlier observations | Hurt |
Surprising findings
- Sequential SGD initialized from the intermediate solution gives a similar improvement to sequential VI, suggesting the gain is not Bayesian belief sharing.
- The martingale posterior ensemble outperforms the initial BMA, contradicting Bayesian predictive coherence and suggesting a possible practical gain from retraining on imputed labels.
Limitations
- Experiments use synthetic regression and CIFAR/IMDB with replicated settings from Izmailov et al. (2021b); generality across architectures is limited.
- Sequential experiments use only a single retraining step or few splits, so estimation error is intentionally minimized and results may not transfer to longer sequences.
- Diagonal Laplace and VI tuned with beta<1 were used, so results depend on hyperparameter choices and cannot separate inference error from model misspecification.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 IMDB synthetic regression
Benchmarks: CIFAR-10 IMDB synthetic regression
Metrics: accuracy log-likelihood expected calibration error BALD score predictive log-likelihood
Baselines: HMC SGD Laplace SWAG deep ensembles
Builds on: martingale posterior Izmailov et al. 2021 BNN posteriors benchmark mean-field variational inference SWAG SGLD
Closest papers by meaning
- On the Expressiveness of Approximate Inference in Bayesian Neural Networks (NeurIPS 2020 · similarity 0.92)
- Transformers Can Do Bayesian Inference (ICLR 2022 · similarity 0.89)
- Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (ICML 2020 · similarity 0.89)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.87)
- Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models (CVPR 2024 · similarity 0.86)
- Position: The Future of Bayesian Prediction Is Prior-Fitted (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.