Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Converts a pre-trained DNN into an ensemble of Bayesian networks post-hoc by replacing normalization layers with Bayesian normalization layers (Gaussian-perturbed scale) and fine-tuning only those parameters for a few epochs.
Problem
Deep neural networks often give unreliable uncertainty estimates, while Bayesian neural networks are unstable and costly to train at scale, and deep ensembles are computationally expensive.
Approach
Replace batch/layer/instance normalization layers with Bayesian Normalization Layers that multiply the learnable scale by (1+epsilon) with Gaussian noise, initialize from the pre-trained weights, and fine-tune only the normalization parameters for a few epochs; train several copies with a class-weighted cross-entropy term to promote diversity.
Key results
- On CIFAR-10 with ResNet-50, ABNN reaches 95.0% accuracy, 96.5 AUPR and 17.5 FPR95, versus 96.0% accuracy and 15.5 FPR95 for Deep Ensembles, at 2.0 vs 6.8 training hours.
- On MUAD, ABNN reaches 61.96 mIoU and 21.68 FPR95, outperforming Deep Ensembles (58.29 mIoU, 37.60 FPR95).
- On ImageNet with ViT, ABNN achieves 80.6% accuracy and 4.32 ECE, versus 81.7% and 13.5 ECE for Deep Ensembles.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ABNN instead of Deep Ensembles | ResNet-50 · ImageNet | accuracy: 79.2 -> 79.5 | Helped |
| Use ABNN instead of Deep Ensembles | ViT · ImageNet | FPR95: 60.3 -> 55.1 | Helped |
| Use ABNN instead of Deep Ensembles | DeepLabv3+ ResNet-50 · MUAD | mIoU: 58.29 -> 61.96 | Helped |
| Use ABNN instead of Deep Ensembles | DeepLabv3+ ResNet-50 · MUAD | FPR95: 37.60 -> 21.68 | Helped |
| Use ABNN instead of Deep Ensembles | DeepLabv3+ ResNet-50 · StreetHazards | mIoU: 55.59 -> 53.82 | Hurt |
| Use ABNN instead of Deep Ensembles | DeepLabv3+ ResNet-50 · BDD-Anomaly | FPR95: 28.55 -> 29.01 | Hurt |
| Discard the loss term E (ablation) | ABNN · image classification experiments | performance | Hurt |
| Discard multi-mode training (ablation) | ABNN · image classification experiments | uncertainty quantification | Hurt |
Surprising findings
- ABNN does not reach the level of diversity of Deep Ensembles, though it departs from a single local minimum.
- Standard deviation across ABNN runs is about five times larger than that of a single model, indicating reduced stability.
Limitations
- ABNN exhibits less diversity than Deep Ensembles.
- Inherits training instabilities of the underlying DNN; run-to-run variance is about five times that of a single model.
- Not compared against Laplace on segmentation because adapting DNNs for Laplace is not straightforward.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet StreetHazards BDD-Anomaly MUAD SVHN Describable Textures
Benchmarks: CIFAR-10 CIFAR-100 ImageNet StreetHazards BDD-Anomaly MUAD
Metrics: accuracy NLL ECE AUPR AUC FPR95 mIoU
Baselines: Deep Ensembles BatchEnsemble MIMO Laplace LP-BNN MC Dropout MCBN TRADI
Builds on: Bayesian neural networks (variational inference) Deep Ensembles Laplace approximation Fort et al. class-weighted diversity
Closest papers by meaning
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (NeurIPS 2017 · similarity 0.90)
- Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (ICML 2020 · similarity 0.89)
- SURE: SUrvey REcipes for building reliable and robust deep networks (CVPR 2024 · similarity 0.87)
- Enhancing Uncertainty Estimation and Interpretability with Bayesian Non-negative Decision Layer (ICLR 2025 · similarity 0.87)
- A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors (ICLR 2024 · similarity 0.87)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.