Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Variational Bayesian Pseudo-Coreset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Variational Bayesian Pseudo-Coreset (VBPC), which uses last-layer variational inference with a Gaussian likelihood to get closed-form coreset posteriors, enabling memory-efficient bilevel training of pseudo-coresets and single-forward-pass Bayesian model averaging.
Problem
Bayesian pseudo-coresets for BNNs require expert trajectories, stop-gradient tricks that hurt performance, and SGMCMC sampling at inference, all of which are memory- and compute-heavy on large datasets.
Approach
Treat only the last layer as random and use natural-gradient VI with a Gaussian likelihood for the pseudo-coreset so the inner VI solution is closed-form; the outer dataset-VI loss uses a probit-style softmax approximation of expected log-likelihood, and a model pool plus low-rank (Woodbury/Weinstein-Aronszajn) computations cut memory from O(h^2) to O(n_hat^2).
Key results
- On CIFAR10 with 10 ipc, VBPC reaches 69.8% ACC and 0.89 NLL versus 62.3% and 1.31 for the best BPC baseline (FBPC).
- On CIFAR100 with 1 ipc, VBPC reaches 38.4% ACC versus 24.0% for BPC-CD.
- Memory-efficient training uses 272.9 MB versus 542.9 MB for naive training on CIFAR10 10 ipc and reduces time to 9.9 s/100 steps from 54.0 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VBPC instead of the strongest listed BPC baseline | Bayesian model averaging on benchmark datasets, 1 ipc · MNIST | ACC: 93.4$\spm{0.1}$ -> 96.7$\spm{0.4}$ | Helped |
| Use VBPC instead of the strongest listed BPC baseline | Bayesian model averaging on benchmark datasets, 1 ipc · Fashion-MNIST | NLL: 1.81$\spm{0.03}$ -> 0.47$\spm{0.03}$ | Helped |
| Use VBPC instead of the strongest listed BPC baseline | Bayesian model averaging on benchmark datasets, 1 ipc · CIFAR10 | ACC: 46.9$\spm{0.2}$ -> 55.1$\spm{0.3}$ | Helped |
| Use VBPC instead of the strongest listed BPC baseline | Bayesian model averaging on benchmark datasets, 1 ipc · CIFAR100 | NLL: 3.76$\spm{0.11}$ -> 2.47$\spm{0.04}$ | Helped |
| Use VBPC instead of the strongest listed BPC baseline | Bayesian model averaging on benchmark datasets, 1 ipc · Tiny-ImageNet | ACC: 10.1$\spm{0.7}$ -> 23.1$\spm{0.2}$ | Helped |
| Use VBPC instead of BPC-CD | CIFAR10-C evaluation with Gaussian Blur corruption · CIFAR10-C, Gaussian Blur | ACC: 41.4$\spm{0.7}$ -> 59.3$\spm{0.9}$ | Helped |
| Use VBPC instead of BPC-CD | CIFAR10 evaluation with Conv-NN architecture · CIFAR10, Conv-NN | ACC: 30.1$\spm{4.4}$ -> 58.4$\spm{0.8}$ | Helped |
| Use memory-efficient loss computation instead of naive training computation (ablation) | VBPC training on CIFAR10 10 ipc · CIFAR10, 10 ipc | training memory: 542.9 -> 272.9 MB | Helped |
| Use memory-efficient loss computation instead of naive training computation (ablation) | VBPC training on CIFAR10 10 ipc · CIFAR10, 10 ipc | training time (sec/100 steps): 54.0 -> 9.9 | Helped |
| Compute BMA without explicitly forming the full covariance (ablation) | VBPC inference on CIFAR10 10 ipc · CIFAR10, 10 ipc | BMA memory: 542.9 -> 268.9 MB | Helped |
Surprising findings
- Under dataset-distillation evaluation, VBPC pseudo-coresets evaluated with FRePo's method are weaker on CIFAR100 with 1 ipc than RCIG (38.4 vs 39.3 ACC), the only setting where VBPC is not best.
Limitations
- Only the last layer is treated as Bayesian, which may limit uncertainty capture for larger models and complex tasks.
- Gaussian likelihood is used for the coreset VI as a detour for closed-form solutions, rather than the softmax likelihood, and the dataset VI uses a mean-field approximation for expected log-softmax.
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR10 CIFAR100 Tiny-ImageNet CIFAR10-C
Benchmarks: MNIST Fashion-MNIST CIFAR10 CIFAR100 Tiny-ImageNet CIFAR10-C
Metrics: ACC NLL
Baselines: BPC-rKL BPC-fKL FBPC BPC-CD FRePo RCIG
Builds on: Bayesian Pseudo-Coreset FRePo Bayesian Learning Rule last-layer variational inference Function-space Bayesian Neural Network
Closest papers by meaning
- Black-box coreset variational inference (NeurIPS 2022 · similarity 0.90)
- Deterministic Variational Inference for Robust Bayesian Neural Networks (ICLR 2019 · similarity 0.89)
- Variational Bayesian Last Layers (ICLR 2024 · similarity 0.88)
- VIKING: Deep variational inference with stochastic projections (NeurIPS 2025 · similarity 0.88)
- Bayesian Online Natural Gradient (BONG) (NeurIPS 2024 · similarity 0.86)
- Variational Learning of Fractional Posteriors (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.