Research map / Large language models / Reasoning and chain of thought
Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes minimal algorithmic tasks modeling combinational and exploratory creativity, showing multi-token prediction (teacherless training, diffusion) beats next-token prediction on algorithmic creativity and that input-level seed-conditioning rivals temperature sampling.
Problem
Language models' creativity in open-ended tasks (diverse, original, coherent outputs) is hard to measure in the wild; next-token prediction and temperature sampling may limit it, and this needs a controlled test-bed.
Approach
Design four minimal algorithmic tasks (sibling discovery, triangle discovery, circle and line construction) with an objective novelty metric, compare next-token vs teacherless multi-token training and diffusion, and inject randomness into the input (seed-conditioning) instead of only at the output (temperature sampling).
Key results
- Multi-token prediction gives nearly 5x algorithmic creativity over next-token prediction on the discovery datasets for Gemma v1 2B.
- Seed-conditioning with greedy decoding yields non-trivial algorithmic creativity comparable to or better than temperature sampling for Transformers.
- For the larger model, next-token prediction memorizes training data substantially more than multi-token training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-token prediction instead of next-token prediction | Gemma v1 (2B) · All four algorithmic tasks | algorithmic creativity: ~5x for the discovery datasets | Helped |
| Use diffusion instead of next-token training | GPT-2 (86M)-sized models · Circle Construction, Line Construction, and Triangle Discovery | algorithmic creativity | Helped |
| Use diffusion instead of next-token training | GPT-2 (86M)-sized models · Sibling Discovery | algorithmic creativity | Hurt |
| Use teacherless training instead of next-token training | smaller Transformers · Algorithmic tasks | algorithmic creativity | No clear effect |
| Use teacherless training instead of next-token training | larger models · Algorithmic tasks | memorization | Helped |
| Add seed-conditioning | Transformers · Algorithmic tasks | algorithmic creativity | Helped |
| Add seed-conditioning | Transformers with greedy decoding · Algorithmic tasks | algorithmic creativity | Helped |
| Increase seed length (ablation) | Next-token and multi-token Transformers · Algorithmic tasks | algorithmic creativity | Helped |
| Add a seed prefix rather than use a null prefix (ablation) | Transformers at a fixed temperature · Algorithmic tasks | algorithmic creativity | Helped |
| Add seed-conditioning | smaller Transformers · Sibling Discovery | algorithmic creativity | No clear effect |
| Add seed-conditioning (ablation) | Diffusion models · Algorithmic tasks | algorithmic creativity | No clear effect |
| Reverse the token order (ablation) | Next-token training · Sibling Discovery | algorithmic creativity | Helped |
Surprising findings
- Seed-conditioning produces diverse outputs even with deterministic greedy decoding, despite seeds being arbitrarily paired with training points.
- Teacherless training on reversed token order achieves even higher creativity than all other training methods tested.
Limitations
- Tasks are minimal caricatures; success does not guarantee success on real creative tasks.
- Teacherless training is hard to optimize for small models; triangle discovery is challenging.
- Seed-conditioning gains for GPT-2 depend on top-K sampling settings and it requires special training.
Experimental setup
Datasets: Sibling Discovery (synthetic) Triangle Discovery (synthetic) Circle Construction (synthetic) Line Construction (synthetic) XSUM CNN/DailyMail
Benchmarks: Sibling Discovery Triangle Discovery Circle Construction Line Construction XSUM CNN/DailyMail
Metrics: algorithmic creativity (novel, coherent, unique fraction) memorization fraction Self-BLEU ROUGE
Baselines: next-token prediction temperature sampling null conditioning
Builds on: teacherless training score entropy discrete diffusion path-star task temperature sampling
Closest papers by meaning
- The Pitfalls of Next-Token Prediction (ICML 2024 · similarity 0.90)
- Auto-Regressive Next-Token Predictors are Universal Learners (ICML 2024 · similarity 0.88)
- How reinforcement learning after next-token prediction facilitates learning (ICLR 2026 · similarity 0.87)
- Reinforcement Mid-Training (ICLR 2026 · similarity 0.87)
- Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers (NeurIPS 2025 · similarity 0.87)
- On the Bias of Next-Token Predictors Toward Systematically Inefficient Reasoning: A Shortest-Path Case Study (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.