Research map / Probabilistic and scientific modeling / Variational and generative sampling
Latent Thought Models with Variational Bayes Inference-Time Computation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Latent Thought Models (LTMs), a Transformer language model with explicit per-sequence latent thought vectors under a Gaussian prior, trained by dual-rate variational Bayes (fast inference-time optimization of latents, slow decoder updates), and studies scaling along inference steps and latent count.
Problem
Language models lack explicit latent thought variables with inference-time computation, and existing scaling laws focus on data and parameters, so sample efficiency under limited data is a concern.
Approach
Treat per-sequence latent thought vectors as local variational parameters optimized by fast gradient steps (Adam, high learning rate) inside classical variational Bayes, alternating with slow global decoder updates, so inference-time compute becomes a scaling axis that improves sample efficiency.
Key results
- LTM-Large (76M, 64 inference steps) reaches 3.05 validation perplexity on OWT after about 3B tokens, and LTM-Medium matches GPT-2-Large's perplexity (10.95 vs 11.5) at equal training FLOPs with about 6.7% of its parameters.
- On zero-shot perplexity, LTM-Large gets 4.43 on PTB and 3.66 on WikiText versus GPT-2-Large's 161.33 and 30.09 (upper bounds from the table).
- On MAUVE conditional generation, LTM-Large scores 0.974 with multinomial sampling, above GPT-2-Medium nucleus (0.955) and SEDD standard (0.957).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LTM-Large instead of the strongest listed baseline | zero-shot language modeling · PTB, WikiText, LM1B, LAMBADA, AG News, PubMed, and Arxiv validation | perplexity | Helped |
| Use LTM-Large instead of SEDD Standard | conditional generation · OWT conditional completion | MAUVE: 0.957 -> 0.974 | Helped |
| Use LTM-Large instead of GPT-2-Large | unconditional generation · unconditional generation | samples/s: 0.014 -> 0.08 | Helped |
| Use LTM-Large instead of GPT-2-Large | unconditional generation · unconditional generation | Gen PPL: 60.4 -> 87.1 | Hurt |
| Use LTM-Large instead of GPT-2-Large | unconditional generation · unconditional generation | Entropy: 5.71 -> 5.61 | Hurt |
| Increase latent thought vectors from 24 to 96 | LTM · validation perplexity across model sizes and inference-step configurations | perplexity | Helped |
| Increase inference steps from 16 to 128 | LTM · validation perplexity | perplexity | Helped |
| Increase model size at fixed latent size and inference steps | LTM · validation perplexity | perplexity | No clear effect |
| Increase inference steps | LTM training · training compute regimes | sample efficiency | Helped |
| Increase latent size to 48 or 96 alongside more inference steps | LTM training · training compute regimes | sample efficiency | Helped |
| Increase LTM model size | LTM · GSM8K zero-shot and few-shot | pass@5 accuracy | Helped |
| Increase latent thought vectors to 192 | LTM-Large · GSM8K | pass@5 accuracy | Helped |
Surprising findings
- The VAE baseline with amortized inference suffered severe posterior collapse, producing repetitive samples (Gen PPL 1.1, entropy 1.83).
- LTM-Large performance is nearly unchanged under greedy decoding on MAUVE (0.972 vs 0.974), suggesting a highly concentrated per-token distribution given latents.
Limitations
- Uses a simple isotropic Gaussian prior; learnable structured prior and reward/verifier in latent space are left for future work.
- Finite-step fast learning may bias the ELBO relative to maximum likelihood when local inference does not converge.
- Perplexity results are at small scale (GPT-2 scale, 38M-76M parameters), and the few-shot arithmetic capability was shown only at this scale.
Experimental setup
Datasets: OpenWebText Penn Tree Bank WikiText LM1B LAMBADA AG News PubMed Arxiv GSM8K
Benchmarks: OpenWebText validation perplexity PTB WikiText LM1B LAMBADA AG News PubMed Arxiv GSM8K MAUVE conditional generation unconditional Gen PPL and entropy
Metrics: perplexity pass@5 MAUVE generative perplexity token-level entropy samples per second
Baselines: GPT-2-Medium GPT-2-Large AR (110M) SEDD MDLM MD4 VAE with amortized inference Langevin sampling
Builds on: variational autoencoder evidence lower bound (ELBO) variational inference dynamic evaluation GPT-2 MDLM
Closest papers by meaning
- LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning (ICLR 2026 · similarity 0.89)
- Latent Thoughts Tuning: Bridging Context and Reasoning with Fused Information in Latent Tokens (ICML 2026 · similarity 0.89)
- Think Thrice Before You Act: Progressive Thought Refinement in Large Language Models (ICLR 2025 · similarity 0.88)
- PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space (ICML 2026 · similarity 0.87)
- Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in its Latent Thoughts (ICLR 2026 · similarity 0.87)
- Large Language Bayes (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.