Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Reparameterized Importance Sampling for Robust Variational Bayesian Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Reparameterized Importance Sampling (RIS), which approximates the optimal importance proposal for first-moment propagation in mean-field variational BNNs and combines it with the reparameterization trick to reduce gradient variance.
Problem
Mean-field variational inference for Bayesian neural networks suffers from Monte Carlo sampling problems: samples lie far from the most probable weights and the stochastic gradients have high variance, slowing or preventing convergence.
Approach
Approximates the optimal importance-sampling proposal (proportional to |f(w)|q(w)) with a Gaussian whose mean and variance are obtained from a Taylor-linearized closed form, then expresses proposal samples as a reparameterized transform of exogenous noise so gradients flow to variational parameters, reducing variance in feed-forward first-moment estimates.
Key results
- On CIFAR-10 with ResNet-20, RIS reaches 87.37 ± 0.26% accuracy versus 83.56 ± 0.45% for tempered MFVI with Monte Carlo sampling.
- On MNIST with LeNet, RIS achieves 99.56 ± 0.03% versus 99.24 ± 0.12% for Monte Carlo sampling.
- RIS with 10 MC samples reaches 87.07 ± 0.27% on CIFAR-10 ResNet-20, while 100-sample MFVI reaches 85.38 ± 0.35%; HMC (90.02%) remains higher than RIS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Monte Carlo sampling with RIS | Bayesian ResNet20 · CIFAR-10 | accuracy: 83.56 ± 0.45 -> 87.37 ± 0.26 | Helped |
| replace Monte Carlo sampling with RIS | Bayesian ResNet56 · CIFAR-10 | accuracy: 84.16 ± 0.38 -> 88.25 ± 0.14 | Helped |
| replace Monte Carlo sampling with RIS | Bayesian ResNet20 · CIFAR-100 | accuracy: 52.47 ± 1.46 -> 55.62 ± 0.82 | Helped |
| replace Monte Carlo sampling with RIS | Bayesian ResNet56 · CIFAR-100 | accuracy: 54.62 ± 1.05 -> 58.75 ± 0.72 | Helped |
| replace Monte Carlo sampling with RIS | Bayesian LeNet · MNIST | accuracy: 99.24 ± 0.12 -> 99.56 ± 0.03 | Helped |
| replace MFVI (tempered) with RIS at 10 samples | ResNet-20 · CIFAR-10 | accuracy: 82.74 ± 0.52 -> 87.07 ± 0.27 | Helped |
| replace MFVI (tempered) with RIS at 100 samples | ResNet-20 · CIFAR-10 | accuracy: 85.38 ± 0.35 -> 87.74 ± 0.23 | Helped |
| use RIS instead of MFVI (tempered) | ResNet-20 · CIFAR-10 | accuracy: 83.56 ± 0.45 -> 87.37 ± 0.26 | Helped |
| use RIS instead of SWAG | ResNet-20 · CIFAR-10 | accuracy: 86.80 ± 0.10 -> 87.37 ± 0.26 | Helped |
| use RIS instead of VOGN | ResNet-20 · CIFAR-10 | accuracy: 85.36 ± 0.25 -> 87.37 ± 0.26 | Helped |
| use RIS instead of GLM | ResNet-20 · CIFAR-10 | accuracy: 84.35 ± 0.18 -> 87.37 ± 0.26 | Helped |
| use RIS instead of Adversarial Sampling | ResNet-20 · CIFAR-10 | accuracy: 86.33 ± 0.45 -> 87.37 ± 0.26 | Helped |
Surprising findings
- Reported HMC accuracy (90.02%) exceeds RIS (87.37%) on CIFAR-10 ResNet-20, so RIS does not beat all baselines.
Limitations
- Method is specific to Gaussian mean-field approximating families.
- Still relies on a cold posterior strategy to avoid collapse to the prior.
- Proposal approximation relies on Taylor linearization and a CLT-based Gaussian assumption without rigorous theoretical guarantees.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100
Benchmarks: MNIST CIFAR-10 CIFAR-100
Metrics: classification accuracy optimization trace (accuracy and KL loss vs wall clock) aleatoric and epistemic uncertainty on OOD data
Baselines: MFVI (tempered) SWAG VOGN GLM Adversarial Sampling HMC
Builds on: Bayes by Backprop Deterministic Variational Inference local reparameterization trick cold posterior
Closest papers by meaning
- Deterministic Variational Inference for Robust Bayesian Neural Networks (ICLR 2019 · similarity 0.90)
- VIKING: Deep variational inference with stochastic projections (NeurIPS 2025 · similarity 0.89)
- Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks (ICLR 2025 · similarity 0.88)
- Reparameterization invariance in approximate Bayesian inference (NeurIPS 2024 · similarity 0.88)
- Training Bayesian Neural Networks with Sparse Subspace Variational Inference (ICLR 2024 · similarity 0.87)
- Parameter Expanded Stochastic Gradient Markov Chain Monte Carlo (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.