Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
VIKING: Deep variational inference with stochastic projections
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VIKING, a fully-correlated variational family for overparametrized networks that splits weight-space uncertainty into the kernel and image of the empirical Fisher-Rao/loss-Jacobian, optimized with a stochastic alternating-projections sampler.
Problem
Mean-field variational inference performs poorly on overparametrized deep networks because its independence assumptions ignore the parameter redundancy (reparametrization manifolds), causing unstable training, poor predictions and miscalibration.
Approach
Build a Gaussian posterior whose covariance is sigma_ker^2 times a projection onto the kernel of the per-datum loss Jacobian plus sigma_im^2 times its orthogonal complement, so only two scalars are learned; samples are produced by a stochastic extension of alternating projections (mini-batch, with noise parameter gamma) and the ELBO is optimized with matrix-free CG.
Key results
- On CIFAR-10 VIKING reaches 0.877 accuracy and 0.041 ECE vs 0.835 accuracy and 0.086 ECE for IVON (three runs).
- On SVHN VIKING achieves the best NLL (0.177) and ECE (0.028) among the listed single-model methods.
- On Imagenette with ResNet-34 (21.8M params), VIKING reaches 0.887 accuracy and 0.403 NLL, beating IVON's 0.876 and 0.656, but IVON has better ECE (0.069 vs 0.077).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use full VIKING ELBO optimization rather than post-hoc tuning of variational parameters only (ablation) | VIKING on Fashion MNIST · Fashion MNIST | accuracy | Helped |
| Add stochastic noise to alternating projections (intermediate gamma rather than no noise, gamma = 1.0) (ablation) | VIKING on Fashion MNIST · Fashion MNIST | test accuracy/generalization | Helped |
| Use stochastic alternating projections rather than only projecting onto the current batch kernel (gamma = 0.0) (ablation) | VIKING on Fashion MNIST · Fashion MNIST | test accuracy/generalization | Helped |
| Warm up with maximum-likelihood training before ELBO optimization (ablation) | VIKING on Fashion MNIST · Fashion MNIST | ELBO training performance | Helped |
| Use full maximum-likelihood convergence for warmup before ELBO optimization (ablation) | VIKING on Fashion MNIST · Fashion MNIST | ELBO training performance | Hurt |
| Replace MAP with VIKING | LeNet on MNIST · MNIST | accuracy: 0.986 -> 0.991 | Helped |
| Replace MAP with VIKING | Small ResNet on SVHN · SVHN | ECE: 0.055 -> 0.028 | Helped |
| Replace MAP with VIKING | Small ResNet on CIFAR-10 · CIFAR-10 | ECE: 0.075 -> 0.041 | Helped |
| Replace IVON with VIKING | ResNet-34 on Imagenette · Imagenette | NLL: 0.656 -> 0.403 | Helped |
| Replace IVON with VIKING | ResNet-34 on Imagenette · Imagenette | ECE: 0.069 -> 0.077 | Hurt |
| Use VIKING rather than IVON | Classifier on SVHN and CIFAR-10 · SVHN and CIFAR-10 | ECE | Mixed |
Surprising findings
- Naively reusing one projected sample through training fails; the stochastic alternating projections are described as essential to training successfully.
- Deep ensembles achieve the best predictive accuracy but not the best calibration metrics.
Limitations
- Computational overhead from projection (CG iterations require backward passes per sample).
- Conjugate gradient solves are numerically brittle, requiring reorthogonalization.
- Posterior is restricted to only two scalar variance parameters, which is simple and may be too restrictive.
Experimental setup
Datasets: MNIST Fashion MNIST SVHN CIFAR-10 Imagenette EMNIST CIFAR-100 KMNIST
Benchmarks: MNIST Fashion MNIST SVHN CIFAR-10 Imagenette toy sinusoid regression OOD detection (MNIST, FMNIST, SVHN, CIFAR-10 with EMNIST/KMNIST/CIFAR-100)
Metrics: accuracy confidence NLL ECE MCE AUROC
Baselines: MAP post-hoc loss-projected method IVON SWAG Last Layer LA SGLD Deep ensemble
Builds on: IVON post-hoc posterior approximation via alternating projections Fisher-Rao metric kernel characterization of reparametrizations SWAG Laplace approximation
Closest papers by meaning
- On the Expressiveness of Approximate Inference in Bayesian Neural Networks (NeurIPS 2020 · similarity 0.91)
- Walsh-Hadamard Variational Inference for Bayesian Deep Learning (NeurIPS 2020 · similarity 0.91)
- Generalized Variational Inference via Optimal Transport (AAAI 2024 · similarity 0.89)
- Globally Convergent Variational Inference (NeurIPS 2024 · similarity 0.89)
- Reparameterized Importance Sampling for Robust Variational Bayesian Neural Networks (ICML 2024 · similarity 0.89)
- Variational Learning of Fractional Posteriors (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.