Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Bayesian Low-Rank Learning (Bella): A Practical Approach to Bayesian Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Bella, which approximates a Bayesian neural network posterior by running SVGD or deep ensembles over low-rank (LoRA-style) perturbations of a shared pre-trained weight matrix, cutting trainable parameters by orders of magnitude.
Problem
Full-parameter Bayesian methods (deep ensembles, SVGD) are computationally infeasible for large pre-trained models, limiting practical Bayesian deep learning for robustness and uncertainty.
Approach
Represent each ensemble/SVGD particle as a low-rank perturbation B_i A_i of a frozen pre-trained weight matrix and run joint SVGD over the low-rank factors, so many particles (100+) fit in the memory of one full model, with an RBF kernel computed efficiently via trace identities.
Key results
- Bella SVGD with 5 particles reaches 93.61% on CAMELYON17 vs 93.98% for full SVGD baseline and 90.25% for the single model, with about 5x smaller storage (439 MB vs 2222 MB).
- Bella can scale to over 100 particles (36.86M trainable params) in 5.19 GB GPU memory, whereas full SVGD runs only up to n=40 on a 48 GB A6000.
- On DomainNet, Bella SVGD (n=5) reaches 81.41% vs 69.75% for the single model, while full SVGD reaches 83.66%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace full-parameter SVGD particles with low-rank parameter updates | SVGD on CLIP ViT-B/32 · CIFAR-100 | accuracy: 87.03 ± 0.29% -> 87.63 ± 0.46% | Helped |
| replace full-parameter SVGD particles with low-rank parameter updates | SVGD on CLIP ViT-B/32 · CAMELYON17 | accuracy: 93.98 ± 1.23% -> 93.61 ± 1.28% | No clear effect |
| replace full-parameter SVGD particles with low-rank parameter updates | SVGD on CLIP ViT-B/32 · DomainNet | accuracy: 83.66 ± 1.76% -> 81.41 ± 3.06% | Hurt |
| replace full-parameter SVGD particles with low-rank parameter updates | SVGD on CLIP ViT-B/32 · ImageNet | accuracy: 79.36 -> 78.24 | Hurt |
| replace full-parameter SVGD particles with low-rank parameter updates | SVGD on CLIP ViT-B/32 · CIFAR-10-C (average) | accuracy: 86.84 -> 83.77 | Hurt |
| replace full-parameter SVGD particles with low-rank parameter updates | SVGD on CLIP ViT-B/32 · VQA v2 Yes/No | accuracy: 91.59 -> 92.46 | Helped |
| replace full-parameter SVGD particles with low-rank parameter updates | SVGD on CLIP ViT-B/32 · VQA v2 Number | accuracy: 58.69 -> 60.19 | Helped |
| increase low-rank adapter rank from 2 toward 16 (ablation) | Bella on CLIP ViT-B/32 · CAMELYON17 | accuracy: 95.08% -> 95.21% | Helped |
| increase the number of Bella parameter particles (ablation) | Bella on CLIP ViT-B/32 · CIFAR-10 | performance | Helped |
| fine-tune projection layers 0–11 only (ablation) | Bella on CLIP ViT-B/32 · CAMELYON17 | accuracy: 94.89 -> 95.83 | Helped |
| fine-tune FC and projection layers 0–5 rather than all layers (ablation) | Bella on CLIP ViT-B/32 · CAMELYON17 | accuracy: 94.89 -> 92.7 | Hurt |
| fine-tune FC and projection layers 6–11 rather than all layers (ablation) | Bella on CLIP ViT-B/32 · CAMELYON17 | accuracy: 94.89 -> 91.39 | Hurt |
Surprising findings
- Bella SVGD achieves 95.83% on CAMELYON17 in one layer-configuration ablation, reported as surpassing the state of the art, while full SVGD reaches 95.21%.
- Bella models are reported to be less robust than full SVGD and ensembles under FGSM adversarial attacks, the paper notes as an exception to its claims.
Limitations
- Adversarial robustness is weaker than full SVGD and ensembles, which the authors attribute to reduced parameter diversity.
- Low-rank parameter gradients could not be manipulated in DeepSpeed, so LLaVA variants were simulated via multiple end-to-end fine-tunes with different seeds rather than true low-rank SVGD.
- Diversity of the posterior is limited since all particles share the frozen pre-trained base, relying on a mode-connectivity conjecture for justification.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 CAMELYON17 DomainNet ImageNet VQA v2
Benchmarks: CIFAR-10 CIFAR-100 CAMELYON17 DomainNet ImageNet CIFAR-10-C STL-10 VQA v2
Metrics: accuracy expected calibration error maximum calibration error Brier score AUROC predictive entropy mutual information FGSM robustness accuracy exact match
Baselines: Single model Deep ensemble (full) SVGD (full) VBL VI SGLD
Builds on: Stein Variational Gradient Descent LoRA deep ensembles Laplace LoRA
Closest papers by meaning
- Bayesian Exploration of Pre-trained Models for Low-shot Image Classification (CVPR 2024 · similarity 0.87)
- Scalable Bayesian Learning with posteriors (ICLR 2025 · similarity 0.87)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.86)
- Training Bayesian Neural Networks with Sparse Subspace Variational Inference (ICLR 2024 · similarity 0.86)
- Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI (ICML 2024 · similarity 0.86)
- A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.