Research map / Probabilistic and scientific modeling / Variational and generative sampling
Foundation Posteriors for Approximate Probabilistic Inference
Mike Wu, Noah Goodman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove the MLM term (train inference head only) (ablation) | MLI · six toy probabilistic programs | log p(z|x): -3.740 -> -1.538 (Latent) | Improved |
| remove the MLM term (ablation) | MLI · Hierarchical toy program | variance of log IW: 5.162e9 -> 119.2 | Improved |
| add program augmentations to meta-training (ablation) | MLI · held-out Rosenbrock | log p(z|x): -246.6 -> -8.160 | Improved |
| add program augmentations to meta-training (ablation) | MLI · held-out Rosenbrock | variance of log IW: 355.9 -> 15.10 | Improved |
| add variational finetuning (1,000 SVI iterations) after MLI pretraining | MLI · held-out Rosenbrock | log p(z|x): -8.160 -> -5.727 | Improved |
| finetune from random initialization instead of MLI pretrained weights (ablation) | finetuning · held-out Rosenbrock | log p(z|x): -5.727 -> -42.54 | Hurt |
| initialize transformer backbone with CodeBERT instead of RoBERTa (ablation) | MLI · six toy programs | variance of log IW: 131.3 -> 69.47 (Multi-level) | Improved |
| initialize backbone with CodeBERT instead of RoBERTa (ablation) | MLI · six toy programs | log p(z|x): -1.827 -> -2.041 (Rosenbrock); -3.268 -> -3.691 (Hierarchical); -3.359 -> -3.054 (Multi-level) | Mixed |
| output scientific notation (a x 10^b) from the inference head (ablation) | MLI inference head · toy programs | training stability and average performance | Hurt |
| finetune the model on all 30 responses instead of zero-shot with k=1 observations | plated MLI · plated IRT | log p(z|x): 0.5174 -> 0.8743 | Improved |
| finetune instead of product of experts at k=5 | plated MLI · plated IRT | log p(z|x): 1.3201 -> 1.7331 | Improved |
| increase minibatch size k from 1 to 5 | zero-shot plated MLI · plated IRT | log p(z|x): 0.5174 -> 1.2824 | Improved |
Closest papers by meaning
- MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMs (ICLR 2026 · similarity 0.88)
- Deep Probabilistic Programming (ICLR 2017 · similarity 0.88)
- Large Language Bayes (NeurIPS 2025 · similarity 0.87)
- MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models (NeurIPS 2024 · similarity 0.86)
- Entropy-Gated Latent Recursion (NeurIPS 2026 · similarity 0.86)
- BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
