Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Understanding the difficulties of posterior predictive estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why simple Monte Carlo estimates of posterior predictive densities have very low signal-to-noise ratio, even under exact inference, and proposes learned importance sampling by optimizing the IW-ELBO as a mitigation.
Problem
Simple Monte Carlo estimation of the posterior predictive density (PPD) under an approximate posterior can have extremely low SNR, yielding biased, unreliable log PPD estimates, which matters for predictive uncertainty and comparing inference methods.
Approach
Derives closed-form SNR expressions for the simple MC PPD estimator in terms of log-normalization constants (or KL divergences), showing SNR decays with train-test mismatch, latent dimensionality, and relative test-set size even under exact inference; then learns an importance sampling proposal by maximizing the IW-ELBO as a surrogate for SNR.
Key results
- Learned IS raises SNR from about 0.04 to about 0.32-0.48 on MovieLens-25M and reduces the flow-vs-Gaussian VI gap from over 20 nats to about 4 nats.
- On logistic regression, simple MC SNR drops from 1.35 in the baseline to 0.03-0.04 under mismatch, dimensionality, or test-size increases, while learned IS keeps SNR in the hundreds.
- Under exact inference on exponential family models, simple MC with 10^6 samples has SNR of 0.03-0.35 and estimates are hundreds of nats off from learned IS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learned importance sampling instead of simple Monte Carlo | PPD estimation under exact inference · Normal, Exp, and Binomial | log PPDq estimate: -1183.98 ± 0.34 -> -774.64 ± 0.00; -559.98 ± 0.16 -> -527.44 ± 0.00; -487.13 ± 1.29 -> -327.41 ± 0.00 | Helped |
| Use learned importance sampling instead of simple Monte Carlo | PPD estimation under approximate inference · Normal, Exp, and Binomial | log PPDq estimate: -1194.32 ± 0.41 -> -775.23 ± 0.00; -576.27 ± 0.14 -> -542.34 ± 0.00; -382.46 ± 0.74 -> -322.66 ± 0.00 | Helped |
| Increase latent dimensionality | Simple MC PPD estimation · Logistic regression, more dimension | SNR: 1.35 ± 0.20 -> 0.04 ± 0.01 | Hurt |
| Increase training-test data mismatch | Simple MC PPD estimation · Logistic regression, more mismatch | SNR: 1.35 ± 0.20 -> 0.03 ± 0.00 | Hurt |
| Increase test data size | Simple MC PPD estimation · Logistic regression, more test data | SNR: 1.35 ± 0.20 -> 0.04 ± 0.01 | Hurt |
| Use learned importance sampling instead of simple Monte Carlo | PPD estimation · Logistic regression, baseline; more dimension; more mismatch; more test data | SNR: 1.35 ± 0.20 -> 645.67 ± 14.99; 0.04 ± 0.01 -> 57.78 ± 1.37; 0.03 ± 0.00 -> 728.63 ± 16.01; 0.04 ± 0.01 -> 802.34 ± 18.37 | Helped |
| Use learned importance sampling instead of simple Monte Carlo | MovieLens 25M PPD estimation · Flow VI and Gaussian VI, K = 106 | log PPDq estimate: -787.27 ± 0.08 -> -777.73 ± 0.01; -811.61 ± 0.13 -> -781.88 ± 0.02 | Helped |
| Use learned importance sampling instead of simple Monte Carlo | MovieLens 25M PPD estimation · Flow VI and Gaussian VI, K = 106 | SNR: 0.04 ± 0.01 -> 0.48 ± 0.29; 0.04 ± 0.01 -> 0.32 ± 0.13 | Helped |
| Increase mismatch, latent dimensionality, or test-data size relative to training data | Simple MC PPD estimation · Linear regression | estimation error | Hurt |
Surprising findings
- SNR of the simple MC PPD estimator can be extremely low even when inference is exact, so the problem is not due to poor posterior approximation.
- Simple MC estimates with a million samples can be off by hundreds of nats, and a flow VI advantage of over 20 nats under simple MC shrinks to about 4 nats under learned IS.
Limitations
- Primary contribution is theoretical; the learned IS approach is an illustrative exploratory method.
- Proposal learning can be computationally expensive when the test data set is large.
- Learned IS requires evaluating the approximate distribution and does not immediately extend to MCMC methods.
Experimental setup
Datasets: MovieLens-25M
Benchmarks: MovieLens-25M
Metrics: log PPD estimate (nats) signal-to-noise ratio estimation error
Baselines: simple Monte Carlo estimator annealed importance sampling
Builds on: importance weighted autoencoder black-box variational inference adaptive importance sampling PSIS-LOO
Closest papers by meaning
- Differentiable Annealed Importance Sampling Minimizes The Jensen-Shannon Divergence Between Initial and Target Distribution (ICML 2024 · similarity 0.85)
- Asymptotically Log-Optimal Bayes-Assisted Confidence Sequences for Bounded Means (NeurIPS 2026 · similarity 0.84)
- Fearless Stochasticity in Expectation Propagation (NeurIPS 2024 · similarity 0.84)
- Quantifying Uncertainty in the Presence of Distribution Shifts (NeurIPS 2025 · similarity 0.84)
- Forward $\chi^2$ Divergence Based Variational Importance Sampling (ICLR 2024 · similarity 0.84)
- Provable Posterior Sampling with Denoising Oracles via Tilted Transport (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.