Research map / Probabilistic and scientific modeling / Variational and generative sampling
Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes twisted Sequential Monte Carlo for probabilistic inference in language models, learning intermediate twist functions via a contrastive twist learning objective, and bidirectional SMC bounds on the log partition function for evaluating inference quality.
Problem
Sampling from unnormalized LM target distributions defined by terminal potentials (RLHF, red-teaming, controlled generation, infilling) is hard because the marginals require intractable future marginalization, and existing inference methods lack tools for evaluating accuracy in both KL directions.
Approach
Learn intermediate twist functions that approximate future expected potential so that SMC resampling focuses on promising partial sequences; train them with a contrastive (EBM-style MLE / KL-to-marginal) objective, and derive bidirectional SMC upper and lower bounds on log Z that sandwich KL divergences in both directions.
Key results
- On toxic story generation, SMC and SIS with the twist-induced proposal achieve tight log Z bounds with orders of magnitude lower K than base-model sampling.
- PPO learns the best proposal for reverse KL KL(q||sigma), while CTL performs best for KL(sigma||q) in toxic generation.
- In infilling, DPG performs best in both KL directions, with CTL comparable to SIXO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the twist-induced proposal instead of the base-model proposal | Twisted SMC and SIS · Toxic story generation | log partition function estimation and sampling efficiency | Helped |
| Resample when using the twist-induced proposal | Twisted SMC · Toxic story generation | log partition function bounds | No clear effect |
| Use SMC resampling with the base-model proposal instead of SIS | Base-model sampling · Toxic story generation with a threshold target | log partition function bounds | Helped |
| Train the proposal with PPO rather than CTL | Toxic story generation · Toxic story generation | forward KL, DKL(q||σ) | Helped |
| Train the proposal with CTL rather than PPO | Toxic story generation · Toxic story generation | reverse KL, DKL(σ||q) | Helped |
| Use CTL rather than the other evaluated methods | Twist-induced proposal · Generation with varied sentiment | KL divergence | Helped |
| Train the proposal with DPG rather than the other evaluated methods | Infilling · Infilling | KL divergence | Helped |
| Use CTL rather than DPG for infilling | Infilling · Infilling | KL divergence | Hurt |
| Train with PPO rather than DPG for infilling | Infilling · Infilling | reverse KL, DKL(σ_oT||q_oT) | Hurt |
| Use approximate positive sampling rather than exact target samples for twist training | CTL · Toxic story generation | KL divergence | Mixed |
Surprising findings
- PPO, despite being mode-seeking, does not necessarily yield the best forward KL; DPG wins on infilling when exact samples are available.
Limitations
- Target posteriors for infilling require exact samples via BDMC, which are not always available.
- Experiments use small models (TinyStories-33M, GPT2-Medium) for toxicity, sentiment and infilling.
Experimental setup
Datasets: Amazon reviews TinyStories
Benchmarks: toxic story generation amazon review sentiment infilling
Metrics: log partition function bounds forward and reverse KL divergence
Baselines: PPO DPG SIXO FUDGE soft Q-learning twist learning SIS
Builds on: sequential Monte Carlo twisted SMC SIXO FUDGE IWAE
Closest papers by meaning
- Ensembling Language Models with Sequential Monte Carlo (NeurIPS 2026 · similarity 0.88)
- Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening (ICML 2026 · similarity 0.88)
- Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo (ICLR 2025 · similarity 0.88)
- Rollout Roulette: A Probabilistic Inference Approach to Inference-Time Scaling of LLMs using Particle-Based Monte Carlo Methods (NeurIPS 2025 · similarity 0.87)
- Amortizing intractable inference in large language models (ICLR 2024 · similarity 0.87)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.