Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies the Bayesian posterior of deep neural networks at scale, showing permutation and scaling weight-space symmetries complicate posteriors, comparing approximate posterior methods via MMD, and releasing a large checkpoint dataset.
Problem
The posterior distribution over weights of deep BNNs is extremely high-dimensional and hard to estimate, and weight-space symmetries (permutation and scaling) may create artificial functionally equivalent modes that obscure it.
Approach
Formalizes permutation and scaling symmetries and shows the Bayesian posterior is a mixture of permuted distributions; introduces the min-mass problem (minimizing L2 norm under scaling, shown log-log convex) to study scaling symmetries; compares posterior approximations via MMD with and without symmetry removal.
Key results
- Deep Ensembles achieve near-zero MMD (0.0) to the reference posterior on all settings, with multi-mode methods outperforming single-mode methods on MMD and accuracy.
- Multi-mode SWAG and Deep Ensembles outperform other methods on epistemic uncertainty metrics (AUPR, FPR95), linking posterior quality to OOD detection.
- No functional collapse observed: in-distribution mutual information between independently trained ResNet-18s on CIFAR-100 has very low variance, and no significant correlation found between ID and OOD mutual information.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-mode estimation with ten independently trained models instead of one mode | Approximate Bayesian posterior methods · MNIST (OptuNet) and CIFAR-100 (ResNet-18) | MMD: MNIST: 15.0 -> 2.1 (Dropout); 18.8 -> 2.8 (BNN); 16.0 -> 1.8 (SWAG); 10.6 -> 1.8 (Laplace). CIFAR-100: 4.5 -> 0.7 (Dropo | Helped |
| Use multi-mode estimation with ten independently trained models instead of one mode | Approximate Bayesian posterior methods · MNIST (OptuNet) and CIFAR-100 (ResNet-18) | accuracy: MNIST: 83.3 -> 92.1 (Dropout); 78.1 -> 86.5 (BNN); 88.3 -> 95.0 (SWAG); 87.9 -> 94.8 (Laplace). CIFAR-100: 69.3 -> 75.5 | Helped |
| Use multi-mode estimation rather than a single mode | Approximate Bayesian posterior methods · MNIST, CIFAR-100, and TinyImageNet | ECE | Mixed |
| Use multi-mode estimation rather than a single mode | Approximate Bayesian posterior methods · MNIST, CIFAR-100, and TinyImageNet | AUPR and FPR95 | Helped |
| Account for permutation symmetries in the posterior | Bayesian neural networks · Two-hidden-neuron perceptron posterior | number of modes | Hurt |
| Decrease the learning rate during training | Networks tracked using successive identifiable-model permutations · MNIST and ResNet-18 | Kendall's tau correlation | Helped |
Surprising findings
- Scaling symmetries persist even with weight decay, so neural networks always seem subject to them, contrary to common belief that L2 regularization removes them.
- In-distribution mutual information does not correlate with OOD mutual information, challenging the intuition that in-distribution diversity indicates OOD detection performance.
Limitations
- Symmetry analysis is formalized mainly for fully connected perceptrons; extension to convolutional residual networks is only sketched.
- Variational BNN could not be trained on TinyImageNet, leaving gaps in comparisons; ResNet-18 Laplace uses last-layer approximations.
- OOD evaluation datasets are not fully representative of real-world OOD detection tasks, as the authors acknowledge.
Experimental setup
Datasets: MNIST CIFAR-100 Tiny-ImageNet Fashion-MNIST SVHN Textures
Benchmarks: MNIST CIFAR-100 Tiny-ImageNet
Metrics: MMD accuracy negative log-likelihood ECE Brier score AUPR FPR95 mutual information
Baselines: Monte Carlo Dropout SWAG Bayes by backpropagation BNN Laplace SGHMC Deep Ensembles
Builds on: Deep Ensembles SWAG Laplace approximation Bayes by backpropagation Monte Carlo Dropout
Closest papers by meaning
- Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (ICML 2020 · similarity 0.90)
- A statistical theory of cold posteriors in deep neural networks (ICLR 2021 · similarity 0.89)
- How Good is the Bayes Posterior in Deep Neural Networks Really? (ICML 2020 · similarity 0.89)
- Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks (ICML 2025 · similarity 0.87)
- The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof (NeurIPS 2024 · similarity 0.87)
- Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.