Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Training Bayesian Neural Networks with Sparse Subspace Variational Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSVI, a fully sparse Bayesian neural network training framework that alternates variational updates within a constant-sparsity random subspace with removal-and-addition of subspace indices guided by weight-distribution SNR criteria.
Problem
Bayesian neural networks are costly to train and infer; existing sparse BNNs either sparsify gradually during training or prune after dense training, so training costs remain high.
Approach
Confine the variational posterior to a sparse axis-aligned subspace of fixed size from the start, and alternately optimize the variational parameters and the subspace via removal (SNR of |theta| under a Gaussian posterior) and gradient-based addition, so sparsity stays constant throughout training.
Key results
- On CIFAR-100 at 90% sparsity, SSVI reaches 75.81% accuracy with 0.10x training FLOPs versus 78.20% for dense BNN.
- On CIFAR-10 at 95% sparsity, SSVI reaches 93.69% accuracy with 0.05x training FLOPs, outperforming mBNN (93.14%) at 96.4% sparsity in the comparison table.
- SSVI outperforms mBNN on OOD detection (e.g., CIFAR-10 train, CIFAR-100 test AUROC 0.86 vs 0.84) and on corrupted-data evaluation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with 90% sparsity using SSVI instead of a dense BNN | ResNet-18 · CIFAR-10 | accuracy: 95.13 -> 94.58 | Hurt |
| Train with 95% sparsity using SSVI instead of a dense BNN | ResNet-18 · CIFAR-10 | Training FLOPs: 1x($1.1\times10^{17}$) -> 0.05x | Helped |
| Train with 90% sparsity using SSVI instead of a dense BNN | ResNet-18 · CIFAR-100 | accuracy: 78.20 -> 75.81 | Hurt |
| Train with 95% sparsity using SSVI instead of a dense BNN | ResNet-18 · CIFAR-100 | Training FLOPs: 1x($1.2\times10^{17}$) -> 0.05x | Helped |
| Use SSVI instead of mBNN | ResNet-18 · CIFAR-10 | accuracy: 93.14 -> 93.74 | Helped |
| Use SSVI instead of mBNN | ResNet-18 · CIFAR-100 | accuracy: 73.42 -> 75.81 | Helped |
| Use SSVI instead of mBNN | ResNet-18 · CIFAR-10 train → CIFAR-100 test | AUROC: 0.84 -> 0.86 | Helped |
| Use SSVI instead of mBNN | ResNet-18 · CIFAR-10 train → SVHN test | AUROC: 0.89 -> 0.93 | Helped |
| Use SNR of absolute weights for removal instead of the other tested removal criteria (ablation) | SSVI · CIFAR-100 | accuracy | Helped |
| Use one-step Monte Carlo for addition rather than multi-step Monte Carlo (ablation) | SSVI · CIFAR-100 | accuracy | No clear effect |
| Initialize newly active variances with the mean of the remaining non-zero σ values rather than a small value (ablation) | SSVI · CIFAR-100 | accuracy | Mixed |
| Increase the initialization mean of σ⁰ from 0.001 to 0.01 (ablation) | VI · CIFAR-100 | accuracy: 0.001 -> 0.01 | Hurt |
Surprising findings
- SSVI at 0.1 density sometimes outperforms full-density VI, which is sensitive to sigma initialization and KL warm-up.
Limitations
- Evaluated only on ResNet-18 with CIFAR-scale image classification datasets.
- Accuracy drops of up to about 3% (CIFAR-100 at 90% sparsity) relative to dense BNN.
- Target sparsity had to be chosen to match prior methods' achieved sparsity for fair comparison.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 SVHN CIFAR-10-C CIFAR-100-C
Benchmarks: CIFAR-10 CIFAR-100 SVHN CIFAR-10-C CIFAR-100-C
Metrics: accuracy loss expected calibration error training FLOPs AUROC AUPR
Baselines: VD SBP AEB mBNN RigL Dense BNN (VI)
Builds on: mean-field variational inference RigL local reparameterization trick mBNN
Closest papers by meaning
- Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee (NeurIPS 2020 · similarity 0.90)
- Deterministic Variational Inference for Robust Bayesian Neural Networks (ICLR 2019 · similarity 0.90)
- Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionality (NeurIPS 2025 · similarity 0.89)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.89)
- Posterior Contraction Rates for sparse Kolmogorov-Arnold Networks in Anisotropic Besov Spaces (NeurIPS 2026 · similarity 0.88)
- An Iterative Min-Min Optimization Method for Sparse Bayesian Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.