Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes

Online Bernstein-von Mises theorem

NeurIPS 2026 · Probabilistic and scientific modeling › Uncertainty and Gaussian processes · approach: Nonparametric and PAC-Bayes Bayesian learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proves a non-asymptotic online Bernstein-von Mises theorem for sequential variational (Gaussian) Bayesian updating over mini-batches, showing accumulated approximation error is negligible once mini-batch size exceeds a dimension-dependent threshold.

Problem

Bayesian online learning repeatedly approximates the updated posterior with a tractable family (here Gaussians via variational projection), and the accumulated approximation error over many mini-batches may not stay small; it is unknown whether the sequentially updated posterior matches the full pos

Approach

Combines a non-asymptotic Laplace/variational analysis of each mini-batch update (with self-concordance-type third and fourth derivative control) with a full-data BvM result, and uses the informative Gaussian prior from previous steps so that the per-step error shrinks with t, yielding an online BvM theorem with total variation error about (p*^3/n)^{1/2} for logistic regression.

Key results

• Proposition for logistic regression: with high probability, uniformly over t, total variation between sequential variational posterior and full posterior is at most C(p*^3/n)^{1/2}, under n >= C(p log^6 ... ∨ p^2 log^4 T).
• Per-step variational error bound improves with t: epsilon_{n,t,TV} = O(t^{-3/2}(p^2/n)^{1/2}), i.e. sharper in t than the naive (p^2/N_t)^{1/2}.
• Numerical experiments suggest that with T = N (one datum per step) the posterior mean has relative efficiency significantly greater than 1 versus batch, so the online BvM theorem is conjectured to fail in that regime.

Surprising findings

• Taking one observation per step (T = N) appears to break asymptotic efficiency, with relative efficiency significantly above 1 in experiments, though this is not formally proven.
• The per-step TV error shrinks with the step index t because the informative prior effectively increases the effective sample size.

Limitations

• Results rely on quadratic approximation valid only when mini-batch size is much larger than dimension p; the T = N regime is not covered.
• Requires strong local regularity conditions (A2) and Gaussian variational family; full verification is given mainly for logistic regression with random Gaussian design.

Experimental setup

Datasets: simulated logistic regression with i.i.d. Gaussian design

Metrics: total variation distance KL divergence relative efficiency

Baselines: batch full posterior implicit SGD batch-means SGD

Builds on: Bernstein-von Mises theorem Laplace approximation self-concordance variational Bayes assumed density filtering

Closest papers by meaning

• Bayesian Online Natural Gradient (BONG) (NeurIPS 2024 · similarity 0.88)
• Recursive PAC-Bayes: A Frequentist Approach to Sequential Prior Updates with No Information Loss (NeurIPS 2024 · similarity 0.86)
• Bernstein–von Mises for Adaptively Collected Data (NeurIPS 2025 · similarity 0.86)
• Online Learning via Learned Latent Bayesian Tracking (NeurIPS 2026 · similarity 0.85)
• Discounted Adaptive Online Learning: Towards Better Regularization (ICML 2024 · similarity 0.85)
• Martingale Posterior Neural Networks for Fast Sequential Decision Making (NeurIPS 2025 · similarity 0.85)