Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Reparameterization invariance in approximate Bayesian inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows Laplace approximations in BNNs fail under reparameterization, explains linearized Laplace's success via GGN pseudo-metric geometry, and proposes a reparameterization-invariant Riemannian (Laplace) diffusion posterior that improves fit over sampled and linearized Laplace.
Problem
Approximate posteriors over Bayesian neural network weights assign different densities to different parametrizations of the same function, and sampled Laplace severely underfits while linearized Laplace works surprisingly well without explanation.
Approach
Views the GGN as a pseudo-metric on weight space whose zero-distance classes are reparameterization equivalence classes, and defines a Riemannian diffusion (Laplace diffusion) on a constant-metric manifold using the neural network predictive; linearized Laplace is shown to be a single-step Euler-Maruyama approximation of this diffusion.
Key results
- On MNIST, Laplace diffusion reaches 0.987 accuracy and 0.042 NLL, versus 0.146 accuracy and 3.812 NLL for sampled Laplace and 0.926 accuracy and 0.306 NLL for linearised Laplace.
- On OOD detection trained on MNIST, Laplace diffusion gets AUROC 0.909 on FMNIST versus 0.758 for linearised Laplace and 0.500 for sampled Laplace.
- On CIFAR-10 Laplace diffusion attains the best NLL (0.345) and accuracy (0.905), though sampled Laplace is better on ECE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Laplace Diffusion instead of Linearised Laplace | LeNet · MNIST test | accuracy: 0.926 -> 0.987 | Helped |
| Use Laplace Diffusion instead of Linearised Laplace | LeNet · MNIST test | NLL: 0.306 -> 0.042 | Helped |
| Use Laplace Diffusion instead of Linearised Laplace | LeNet · FMNIST test | accuracy: 0.862 -> 0.906 | Helped |
| Use Laplace Diffusion instead of Linearised Laplace | LeNet · FMNIST test | NLL: 0.423 -> 0.001 | Helped |
| Use Laplace Diffusion instead of Linearised Laplace | ResNet · CIFAR-10 test | accuracy: 0.863 -> 0.905 | Helped |
| Use Laplace Diffusion instead of Linearised Laplace | ResNet · CIFAR-10 test | NLL: 0.614 -> 0.345 | Helped |
| Use Laplace Diffusion instead of Linearised Laplace | LeNet · MNIST-trained OOD detection suite | AUROC: 0.758 -> 0.909; 0.602 -> 0.625; 0.790 -> 0.929 | Helped |
| Use Laplace Diffusion instead of Linearised Laplace | LeNet · FMNIST-trained OOD detection suite | AUROC: 0.625 -> 0.759; 0.628 -> 0.741; 0.624 -> 0.749 | Helped |
| Use Laplace Diffusion instead of Linearised Laplace | ResNet · CIFAR-10-trained OOD detection suite | AUROC: 0.837 -> 0.851; 0.854 -> 0.862 | Helped |
| Use Laplace Diffusion instead of Sampled Laplace | LeNet · MNIST test | accuracy: 0.146 -> 0.987 | Helped |
| Use Laplace Diffusion instead of Sampled Laplace | LeNet · FMNIST test | accuracy: 0.098 -> 0.906 | Helped |
Surprising findings
- Sampled Laplace with the neural network predictive severely underfits, with accuracy as low as 0.146 on MNIST.
- The linearized Laplace approximation is interpreted as a single-step numerical approximation to a reparameterization-invariant diffusion.
Limitations
- Diffusion sampling increases runtime by a constant factor over standard Laplace.
- Common diagonal or block-diagonal GGN approximations break the reparameterization invariance motivation.
- Theoretical results rely on assumptions such as uniform NTK bounds and Lipschitz Jacobians.
Experimental setup
Datasets: MNIST FashionMNIST CIFAR-10 Rotated-MNIST Rotated-FashionMNIST Rotated-CIFAR EMNIST KMNIST SVHN CIFAR-100
Benchmarks: MNIST FashionMNIST CIFAR-10 Rotated-MNIST Rotated-FashionMNIST Rotated-CIFAR
Metrics: confidence accuracy NLL Brier score ECE MCE AUROC
Baselines: sampled Laplace linearised Laplace SWAG diagonal Laplace last-layer Laplace
Builds on: linearized Laplace approximation Laplace approximation Riemannian Laplace approximation Euler-Maruyama scheme
Closest papers by meaning
- Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling (ICML 2022 · similarity 0.89)
- Riemannian Laplace approximations for Bayesian neural networks (NeurIPS 2023 · similarity 0.88)
- Reparameterized Importance Sampling for Robust Variational Bayesian Neural Networks (ICML 2024 · similarity 0.88)
- VIKING: Deep variational inference with stochastic projections (NeurIPS 2025 · similarity 0.87)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.85)
- Do Bayesian Neural Networks Actually Behave Like Bayesian Models? (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.