Research map / Probabilistic and scientific modeling / Variational and generative sampling
Fearless Stochasticity in Expectation Propagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows EP's moment-matching updates are natural-gradient descent on a variational objective, and proposes EP-η and EP-μ, two Monte Carlo-robust EP variants that work with a single sample and avoid debiasing estimators.
Problem
Expectation propagation's moment-matching updates are biased and unstable when moments are estimated from Monte Carlo samples; prior fixes rely on MVN-specific debiasing and many samples with thinning.
Approach
Reinterprets EP updates as natural-gradient descent on the EP variational objective with respect to mean parameters of surrogate tilted distributions; this yields two variants: EP-η (NGD in natural parameters, unbiased in site parameters) and EP-μ (NGD in mean parameters with explicit step size, bias O(ε²)), both stable with one sample.
Key results
- EP-η and EP-μ typically reach a given accuracy faster than EP and SNEP across four inference tasks, often significantly so, with n_samp=1 and n_inner=1.
- EP-μ bias scales as O(ε²) while expected decrease in L is O(ε), versus EP bias O(α) with equal rate of progress decrease.
- On the cosmic radiation and HLR tasks EP-η/EP-μ consistently reach target accuracy significantly faster than EP and SNEP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EP-η or EP-μ updates instead of EP | EP on synthetic hierarchical logistic regression with MVN prior · Synthetic hierarchical logistic regression (MVN prior) | KL divergence at a given number of NUTS steps | Helped |
| Use EP-η or EP-μ updates instead of EP | EP on political survey hierarchical logistic regression with NIW prior · Political survey hierarchical logistic regression (NIW prior) | KL divergence at a given number of NUTS steps | Helped |
| Use EP-η or EP-μ updates instead of EP | EP on synthetic cosmic radiation model · Synthetic cosmic radiation model | KL divergence at a given number of NUTS steps | Helped |
| Use EP-η or EP-μ updates instead of EP | EP on neural response model · Neural response model | KL divergence at a given number of NUTS steps | Helped |
| Use EP-η or EP-μ instead of SNEP | Approximate inference with MVN prior · Synthetic hierarchical logistic regression (MVN prior) | KL divergence at a given number of NUTS steps | Helped |
| Use EP-η or EP-μ instead of SNEP | Approximate inference with MVN prior · Synthetic cosmic radiation model | KL divergence at a given number of NUTS steps | Helped |
| Use EP-μ instead of EP-η | Proposed EP variants · Probabilistic inference tasks | KL divergence | No clear effect |
| Use NUTS rather than an oracle sampling kernel | EP-η compared with CVI · Hierarchical logistic regression | Wall-clock time | Hurt |
| Use EP-η rather than CVI | Hierarchical logistic regression · Hierarchical logistic regression | Posterior approximation fidelity | Helped |
Surprising findings
- SNEP converges significantly slower than EP and the new variants, consistent with prior findings.
- EP-η with NUTS is much slower in wall-clock time than CVI, though its sample efficiency matches CVI with an oracle sampler.
Limitations
- Wall-clock cost is dominated by the underlying MCMC sampler (NUTS), making EP-η slower than CVI in time despite better sample efficiency.
- Hyperparameter tuning of the underlying sampler remains costly; EP-η/EP-μ do not reduce it.
- Experiments are small scale due to comprehensive hyperparameter searches; the cosmic radiation dataset was unavailable and replaced with synthetic data.
Experimental setup
Datasets: synthetic hierarchical logistic regression data 2018 Cooperative Congressional Election Study survey data GALEX cosmic radiation data (synthetic replacement) V1 complex cell recordings from anesthetised cat
Benchmarks: hierarchical logistic regression (MVN prior) hierarchical logistic regression (NIW prior) cosmic radiation model neural response model (V1 cat) stochastic clutter problem
Metrics: KL divergence to optimum vs NUTS steps (Pareto frontier) bias in lambda wall-clock time Pareto frontier
Baselines: EP (debiased) EP (naive) SNEP CVI
Builds on: expectation propagation power EP stochastic natural gradient EP sampling via moment sharing
Closest papers by meaning
- State Space Expectation Propagation: Efficient Inference Schemes for Temporal Gaussian Processes (ICML 2020 · similarity 0.88)
- Graphical model inference: Sequential Monte Carlo meets deterministic approximations (NeurIPS 2018 · similarity 0.87)
- Black-Box Variational Inference as a Parametric Approximation to Langevin Dynamics (ICML 2020 · similarity 0.86)
- A Stochastic Path Integral Differential EstimatoR Expectation Maximization Algorithm (NeurIPS 2020 · similarity 0.86)
- On the Global Convergence of (Fast) Incremental Expectation Maximization Methods (NeurIPS 2019 · similarity 0.86)
- Understanding the difficulties of posterior predictive estimation (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.