Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Variational Bayesian Last Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes variational Bayesian last layers (VBLLs), a deterministic, sampling-free variational treatment of Bayesian last-layer networks for regression and classification, with optional variational feature learning.
Problem
Uncertainty quantification in deep networks is costly for Bayesian methods; exact Bayesian last-layer marginal likelihood training is not scalable (full-batch, long graphs, overconcentration), and sampling-based SVI has high variance and cost.
Approach
Derive deterministic lower bounds on the marginal likelihood for Bayesian last layers (regression, discriminative and generative classification) that allow mini-batch, sampling-free training at roughly the cost of standard network training, with closed-form predictions via conjugacy and optional collapsed variational inference over features.
Key results
- On CIFAR-10 with Wide ResNet-28-10, D-VBLL reaches 96.4% accuracy versus 95.8% for the DNN and 95.7% for SNGP, and beats SNGP on SVHN AUC (0.969 vs 0.960).
- On the wheel bandit, VBLL achieves cumulative regret 24.44 at delta=0.99 versus 37.77 for the best baseline LinDiagPost.
- On UCI regression, VBLL is best or competitive across datasets but is not uniformly best; e.g. RBF GP and Ensemble win several NLL/RMSE comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use D-VBLL instead of a deterministic DNN | Wide ResNet-28-10 · CIFAR-10 | accuracy: 95.8 ± 0.19 -> 96.4 ± 0.12 | Helped |
| Use D-VBLL instead of SNGP | Wide ResNet-28-10 · CIFAR-10 | SVHN AUC: 0.960 ± 0.004 -> 0.969 ± 0.004 | Helped |
| Use D-VBLL instead of DNN + LL Laplace | Wide ResNet-28-10 · CIFAR-10 | ECE: 0.010 ± 0.001 -> 0.022 ± 0.001 | Hurt |
| Use D-VBLL instead of DNN | Wide ResNet-28-10 · CIFAR-100 | accuracy: 80.4 ± 0.29 -> 80.7 ± 0.03 | Helped |
| Use D-VBLL instead of SNGP | Wide ResNet-28-10 · CIFAR-100 | accuracy: 80.3 ± 0.23 -> 80.7 ± 0.03 | Helped |
| Use D-VBLL instead of SNGP | Wide ResNet-28-10 · CIFAR-100 | ECE: 0.030 ± 0.004 -> 0.040 ± 0.002 | Hurt |
| Use D-VBLL instead of Ensemble | Wide ResNet-28-10 · CIFAR-100 | accuracy: 82.5 ± 0.19 -> 80.7 ± 0.03 | Hurt |
| Use VBLL instead of NeuralLinear | Wheel bandit with Thompson sampling · Wheel bandit (δ = 0.99) | cumulative regret: 52.64 ± 2.04 -> 24.44 ± 0.71 | Helped |
| Add a D-VBLL post-training layer to a trained DNN (ablation) | Wide ResNet-28-10 · CIFAR-10 | accuracy: 95.8 ± 0.19 -> 96.4 ± 0.01 | No clear effect |
| Add a G-VBLL post-training layer to a trained DNN (ablation) | Wide ResNet-28-10 · CIFAR-10 | SVHN AUC: 0.946 ± 0.005 -> 0.729 ± 0.004 | Hurt |
| Add a G-VBLL post-training layer to a trained DNN (ablation) | Wide ResNet-28-10 · CIFAR-100 | NLL: 0.941 ± 0.016 -> 3.026 ± 0.155 | Hurt |
| Add variational feature learning (BBB) to D-VBLL (ablation) | D-VBLL · CIFAR-10 | accuracy: 96.4 ± 0.12 -> 95.9 ± 0.15 | Hurt |
Surprising findings
- Variational BBB-based feature learning was worse than deterministic models on classification, which the authors attribute to possible over-regularization.
- The generative VBLL (G-VBLL) performed substantially worse than the discriminative variant as a post-training method.
Limitations
- Generative classification cannot be computed with quadratic complexity without diagonal covariance assumptions.
- Discriminative classification lower bound is not tight (two Jensen applications) and prediction requires logit sampling.
- Variational feature learning (BBB) results are mixed and sometimes worse than deterministic models, with the regularization weighting unresolved.
Experimental setup
Datasets: UCI regression datasets (Boston, Concrete, Energy, Power, Wine, Yacht) CIFAR-10 CIFAR-100 SVHN IMDB
Benchmarks: UCI regression CIFAR-10 CIFAR-100 SVHN OOD IMDB sentiment wheel bandit
Metrics: NLL RMSE accuracy ECE AUC cumulative regret simple regret
Baselines: GBLL LDGBLL MAP RBF GP Dropout Ensemble SWAG BBB SNGP DNN + LL Laplace NeuralLinear LinDiagPost
Builds on: Bayesian last layer SNGP Bayesian dropout stochastic variational inference deep kernel learning
Closest papers by meaning
- Bayesian Optimization via Continual Variational Last Layer Training (ICLR 2025 · similarity 0.90)
- Variational Bayesian Pseudo-Coreset (ICLR 2025 · similarity 0.88)
- Variational Learning of Fractional Posteriors (ICML 2025 · similarity 0.87)
- Bayesian Online Natural Gradient (BONG) (NeurIPS 2024 · similarity 0.87)
- Enhancing Uncertainty Estimation and Interpretability with Bayesian Non-negative Decision Layer (ICLR 2025 · similarity 0.87)
- Maxitive Donsker-Varadhan Formulation for Possibilistic Variational Inference (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.