Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Variational Linearized Laplace Approximation for Bayesian Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VaLLA, a variational sparse GP (dual RKHS formulation with decoupled inducing points) that keeps the pre-trained DNN output as the predictive mean while approximating LLA's predictive variance with mini-batch stochastic optimization.
Problem
Linearized Laplace Approximation (LLA) uncertainty estimation for pre-trained DNNs is computationally expensive for large training sets and many parameters, and common approximations (KFAC, diagonal GGN) can hurt performance.
Approach
Interpret LLA as a GP and fit a decoupled variational sparse GP in the RKHS dual form, fixing the posterior mean to the DNN output and using separate inducing points for mean and covariance, optimized with mini-batches so training cost is independent of the number of training points; hyperparameters are tuned with an alpha=1 objective plus early stopping.
Key results
- On CIFAR10 ResNets, VaLLA (M=100) achieves the best or second-best NLL and ECE in most cases, with mean rank 1.375 versus 2.0 for Sampled LLA and 2.375 for ELLA.
- On large regression datasets (Year, Airline, Taxi), VaLLA performs best on NLL and CQM but worse on CRPS than the other methods, averaged over 5 seeds.
- On Taxi, early stopping triggers after only 16.6% of the training data has been seen for one seed, reducing training time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VaLLA instead of Sampled LLA | CIFAR10 ResNet models · CIFAR10, ResNet-20/32/44/56 | NLL: 0.231 -> 0.228; 0.217 -> 0.211; 0.200 -> 0.198; 0.185 -> 0.183 | Helped |
| Use VaLLA instead of Sampled LLA | CIFAR10 ResNet models · CIFAR10, ResNet-20/32/44/56 | ACC: 92.5 -> 92.6; 93.5 -> 93.5; 94.0 -> 94.0; 94.4 -> 94.4 | Mixed |
| Use VaLLA instead of Sampled LLA | CIFAR10 ResNet models · CIFAR10, ResNet-20/32/44/56 | ECE: 0.006 -> 0.007; 0.008 -> 0.007; 0.007 -> 0.008; 0.015 -> 0.009 | Mixed |
| Use VaLLA rather than MAP | pre-trained DNN uncertainty estimation · MNIST | NLL | Helped |
| Use VaLLA rather than MAP | pre-trained DNN uncertainty estimation · MNIST | OOD-AUC | Helped |
| Use VaLLA rather than other methods | pre-trained DNN uncertainty estimation · FMNIST | NLL | Helped |
| Use VaLLA rather than other methods | pre-trained DNN uncertainty estimation · FMNIST | OOD-AUC | Helped |
| Use VaLLA rather than other methods | pre-trained DNN uncertainty estimation · Year, Airline and Taxi regression | NLL | Helped |
| Use VaLLA rather than other methods | pre-trained DNN uncertainty estimation · Year, Airline and Taxi regression | CRPS | Hurt |
| Increase VaLLA inducing points relative to ELLA points | toy 1-D regression posterior approximation · Synthetic regression | predictive variance approximation | Helped |
| Use early stopping with a validation set | VaLLA prior-variance optimization · Synthetic regression | predictive variance | Helped |
Surprising findings
- VaLLA's mean rank on CIFAR10 is better than sampled LLA and ELLA despite using only 100 inducing points.
- Maximizing the standard ELBO for prior variance tunes the predictive variance toward zero in VaLLA, which motivated the alpha-divergence objective with alpha=1 and early stopping.
Limitations
- Cubic cost in the number of inducing points, so it cannot scale to very many inducing points.
- Requires a validation set and early stopping to tune the prior variance, which may overfit otherwise and adds training time.
- Computing the NTK and its gradients is costly for large architectures; the layer-by-layer method is difficult for complex DNNs, and ImageNet is deemed infeasible.
Experimental setup
Datasets: UCI Year US flight delay (Airline) Taxi January 2023 MNIST FMNIST CIFAR10
Benchmarks: Year Airline Taxi MNIST FMNIST CIFAR10 corrupted CIFAR10
Metrics: NLL CRPS CQM ECE ACC Brier score OOD-AUC training time
Baselines: LLA LLA KFAC LLA Diag ELLA Sampled LLA GP Subset SNGP MF-VI MoE LLA
Builds on: linearized laplace approximation sparse variational gaussian process decoupled sparse GP ELLA
Closest papers by meaning
- Accelerated Linearized Laplace Approximation for Bayesian Deep Learning (NeurIPS 2022 · similarity 0.94)
- FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learning (NeurIPS 2024 · similarity 0.86)
- New Bounds for Sparse Variational Gaussian Processes (ICML 2025 · similarity 0.86)
- Training Bayesian Neural Networks with Sparse Subspace Variational Inference (ICLR 2024 · similarity 0.85)
- VIKING: Deep variational inference with stochastic projections (NeurIPS 2025 · similarity 0.85)
- Bayesian Optimization via Continual Variational Last Layer Training (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.