Research map / Probabilistic and scientific modeling / Variational and generative sampling
Large Language Bayes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Large Language Bayes, gluing an LLM (generating Stan models from an informal description) to a PPL and approximating the model-averaged posterior via LLM sampling, per-model MCMC, importance-weighted ELBO bounds and SNIS weighting.
Problem
Domain experts often lack the time or expertise to write formal Bayesian models; the paper aims to produce posteriors over target variables directly from an informal problem description plus data.
Approach
Define a joint p(z,x,m|t)=p(m|t)p(z,x|m), with the LLM supplying p(m|t) and the PPL supplying the model likelihood; approximate the posterior by self-normalized importance sampling over LLM-sampled models, using MCMC posteriors and importance-weighted bounds on marginal likelihoods.
Key results
- On the rain problem, 960 of 1024 generated models compiled; the final posterior is dominated by one model predicting 0.793, versus a flat average of 0.573.
- On gold with 150 observations, weight concentrates on one model and the posterior is close to the true density and clearly better than a flat average.
- On polling and city temperature, the final posterior shows little benefit over the flat average, with one model taking nearly all weight in polling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Weight generated models by estimated marginal likelihood instead of taking a flat average | LLB · Rain | predicted rain probability: 0.573 -> 0.793 | Mixed |
| Weight generated models by estimated marginal likelihood instead of taking a flat average | LLB · Polling | posterior prediction | No clear effect |
| Weight generated models by estimated marginal likelihood instead of taking a flat average | LLB · City temperature | posterior prediction | No clear effect |
| Weight generated models by estimated marginal likelihood instead of taking a flat average | LLB · Gold, 150 observations | posterior prediction | Helped |
| Use a larger dataset (150 rather than 30 observations) | LLB · Gold | posterior prediction | Helped |
| State a different prior-bias assumption in the prompt | LLB · Coin | posterior over true coin bias | Mixed |
| Match the Gaussian proposal to MCMC sample mean and variance rather than explicitly optimizing it | Marginal-likelihood estimation · Heterogeneous generated models | inference performance | No clear effect |
Surprising findings
- Gold problem: only 1.2% (30 data) and 0.4% (150 data) of 16,384 generated models compiled and allowed inference.
- Polling: a single simple model captured essentially all posterior weight, which may indicate inaccurate inference.
Limitations
- Inference is run independently per sampled model, which is expensive and practical only for small problems.
- Many generated models fail to compile, with very low yield on the gold problem.
- Truncated distributions in Stan are not renormalized, biasing marginal likelihoods against some models.
Experimental setup
Datasets: synthetic gold density data synthetic rain sequence coin flips (14 of 20) polling data city temperature data
Benchmarks: rain coin polling city temperature gold
Metrics: posterior predictions compared to flat average marginal likelihood weights compile rate
Baselines: flat average of LLM-generated model posteriors
Builds on: Bayesian model averaging importance-weighted variational inference NUTS
Closest papers by meaning
- BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models (ICLR 2025 · similarity 0.90)
- Large Language Models to Enhance Bayesian Optimization (ICLR 2024 · similarity 0.88)
- Reasoning over Uncertain Text by Generative Large Language Models (AAAI 2025 · similarity 0.87)
- Training-Free Bayesianization for Low-Rank Adapters of Large Language Models (NeurIPS 2025 · similarity 0.87)
- Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective (ICML 2024 · similarity 0.87)
- Latent Thought Models with Variational Bayes Inference-Time Computation (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.