Research map / Large language models / Reasoning and chain of thought
Better & Faster Large Language Models via Multi-token Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes training LLMs to predict n future tokens at once with n independent heads on a shared trunk, showing gains on code and generative benchmarks, faster self-speculative decoding, and induction/algorithmic benefits at scale.
Problem
Next-token prediction is sample-inefficient and suffers from teacher-forcing mismatch, latching onto local patterns and missing consequential decisions in generation; the paper asks whether multi-token prediction can improve sample efficiency and generative capability at no cost.
Approach
Use n independent transformer-layer output heads on a shared trunk to predict the next n tokens at each position, with a memory-efficient sequential forward/backward over heads, and reuse the extra heads for self-speculative decoding.
Key results
- On 7B code models trained on 200B tokens, 4-token prediction improves MBPP pass@1 from 30.0 to 33.8 and HumanEval pass@1 from 22.8 to 24.0 over next-token prediction.
- 13B models with 4-token prediction reach HumanEval pass@1 of 15.8 vs 14.1 and MBPP pass@1 of 30.5 vs 26.0 for next-token prediction.
- Self-speculative decoding with 4-token heads yields about 3.0x relative speedup on code and 2.7x on Wikipedia text at greedy decoding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| predict 4 future tokens instead of 1 | 7B code model trained on 200B tokens · MBPP | pass@1: 30.0 -> 33.8 (+3.8%) | Helped |
| predict 4 future tokens instead of 1 | 7B code model trained on 200B tokens · HumanEval | pass@1: 22.8 -> 24.0 (+1.2%) | Helped |
| predict 6 future tokens instead of 1 (ablation) | 7B code model trained on 200B tokens · APPS/Intro | pass@1: 2.8 -> 3.5 (+0.7%) | Helped |
| increase the prediction window from 1 to 4 future tokens (ablation) | 7B code model trained on 200B tokens · MBPP and HumanEval | pass@1: 30.0 -> 33.8; 22.8 -> 24.0 | Mixed |
| use 4 prediction heads for self-speculative decoding | 7B 4-token prediction model · Code decoding | relative speedup: 3.05 | Helped |
| use 8 prediction heads for self-speculative decoding | 7B 8-byte prediction model · Code decoding | relative speedup: 6.39 | Helped |
| replace transformer prediction heads with linear heads (ablation) | 4-token prediction model · MBPP | pass@1: 33.8 -> 33.6 | No clear effect |
| use anticausal rather than parallel transformer heads (ablation) | 4-token prediction model · MBPP | pass@1: 33.8 -> 30.8 | Hurt |
| use causal rather than parallel transformer heads (ablation) | 4-token prediction model · MBPP | pass@1: 33.8 -> 31.9 | Hurt |
| finetune Llama 2 with 4-token prediction (ablation) | Llama 2 · MBPP | pass@1: 39.6 -> 38.3 | No clear effect |
| train with 2-token rather than next-token prediction (ablation) | 7B natural-language model trained on 200B tokens · standard NLP benchmarks | average accuracy | No clear effect |
| train with 4-token rather than next-token prediction (ablation) | 7B natural-language model trained on 200B tokens · standard NLP benchmarks | average accuracy | Hurt |
Surprising findings
- Multi-token prediction hurts on small model sizes (0.3B-0.6B) for code, and shows no gain on standard multiple-choice NLP benchmarks at 7B.
- Finetuning a Llama 2 model with 4-token prediction did not significantly improve over the baseline.
Limitations
- Gains on multiple-choice NLP benchmarks are absent at 7B, and 4-token prediction regresses slightly; optimal n varies by task and must be tuned.
- Multi-token benefit at small scale is negative for code, so gains depend on model size.
- Byte-level and tokenizer-size results depend on specific vocabulary settings; the optimal n is not automatically chosen.
Experimental setup
Datasets: code training corpus (200B tokens, 1T tokens) natural language corpus (200B, 500B tokens) CodeContests CNN/Dailymail Multi-News OrangeSum pn-summary SAMSum ThaiSum WikiSummary XSum GSM8K
Benchmarks: MBPP HumanEval APPS/Intro CodeContests ARC Challenge COPA Hellaswag Natural Questions PIQA SIQA TriviaQA eight summarization benchmarks (ROUGE)
Metrics: pass@1 pass@10 pass@100 ROUGE-L F1 accuracy relative speedup tokens per forward
Baselines: next-token prediction baseline linear heads causal and anticausal head variants Llama 2
Builds on: blockwise parallel decoding speculative decoding Medusa
Closest papers by meaning
- L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models (NeurIPS 2025 · similarity 0.92)
- Beyond Next Token Prediction: Patch-Level Training for Large Language Models (ICLR 2025 · similarity 0.92)
- Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing (ICML 2026 · similarity 0.91)
- Multi-Token Prediction Needs Registers (NeurIPS 2025 · similarity 0.89)
- Think before you speak: Training Language Models With Pause Tokens (ICLR 2024 · similarity 0.89)
- Reinforced Fast Weights via Next-Sequence Prediction (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.