Research map / Architectures and efficiency / Attention and transformers
The Pitfalls of Next-Token Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that teacher-forced next-token training can fail in-distribution on a minimal path-finding (path-star) lookahead task for Transformers and Mamba, and that teacherless training with dummy tokens partially fixes it.
Problem
Whether next-token prediction can learn planning/lookahead tasks: the paper argues the common criticism of snowballing errors conflates autoregressive inference with teacher-forced training, and that teacher-forcing itself can fail to learn an accurate next-token predictor.
Approach
Identifies two teacher-forcing failure mechanisms (the Clever Hans cheat, where revealed ground-truth prefixes allow shortcuts for later tokens, and the indecipherable token, where the first token loses supervision) and tests teacherless training (dummy-token inputs predicting all future tokens) and path reversal as interventions.
Key results
- Standard teacher-forced GPT-Mini, GPT2-Large and Mamba reach only about 1/d accuracy in-distribution on path-star graphs (e.g., ~49% on G_{2,5}, ~19% on G_{5,5}), despite 200k training samples.
- Teacherless training lets GPT2-Large solve most graph tasks (99.9% on G_{2,5}, 99.0% on G_{5,5}), while from-scratch models mostly fail.
- Reversing the target path makes near-perfect accuracy achievable across graphs for all models (e.g., 99.7-100% for GPT-Mini on most topologies).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with teacher forcing rather than a lookahead solution | GPT-Mini, GPT2-Large, and Mamba · Path-star graphs | exact-match accuracy | Hurt |
| Replace ground-truth response prefixes with dummy tokens during training and inference (ablation) | GPT2-Large · G_{2, 5} | exact-match accuracy: 48.9 -> 99.9 | Helped |
| Replace ground-truth response prefixes with dummy tokens during training and inference (ablation) | GPT2-Large · G_{5, 5} | exact-match accuracy: 19.4 -> 99.0 | Helped |
| Replace ground-truth response prefixes with dummy tokens during training and inference (ablation) | GPT2-Large · G_{10, 5} | exact-match accuracy: 10.3 -> 97.8 | Helped |
| Replace ground-truth response prefixes with dummy tokens during training and inference (ablation) | GPT2-Large · G_{2, 20} | exact-match accuracy: 49.2 -> 0.0 | Hurt |
| Replace ground-truth response prefixes with dummy tokens during training and inference (ablation) | GPT2-Large · G_{20, 5} | exact-match accuracy: 3.5 -> 0.0 | Hurt |
| Replace ground-truth response prefixes with dummy tokens during training and inference (ablation) | GPT-Mini · G_{2, 5} | exact-match accuracy: 49.8 -> 99.9 | Helped |
| Replace ground-truth response prefixes with dummy tokens during training and inference (ablation) | GPT-Mini · G_{2, 10} | exact-match accuracy: 49.1 -> 0.0 | Hurt |
| Replace ground-truth response prefixes with dummy tokens during training and inference (ablation) | Mamba · Path-star graphs | exact-match accuracy: 48.5 -> 0.0; 48.7 -> 0.0; 20.2 -> 0.0; 9.3 -> 0.0; 0.0 -> 0.0 | Hurt |
| Reverse the target path (ablation) | GPT2-Large · G_{2, 5} | exact-match accuracy: 48.9 -> 99.9 | Helped |
| Reverse the target path (ablation) | GPT2-Large · G_{5, 5} | exact-match accuracy: 19.4 -> 99.6 | Helped |
| Reverse the target path (ablation) | GPT2-Large · G_{10, 5} | exact-match accuracy: 10.3 -> 99.8 | Helped |
Surprising findings
- Even when trained on 200k samples to 100% training accuracy, teacher-forced models fail in-distribution on a path-finding task that is straightforward when viewed right-to-left.
- Snowballing error is largely absent in this setting; the failure happens at the first token, so it is not primarily an inference-time error accumulation problem.
Limitations
- Results are empirical and conceptual; no formal proof of the failure mechanisms.
- Failure not demonstrated for very large models such as Llama2 or Mistral, and workarounds (pretraining, parallel solutions) may make the path-star task solvable.
- Unclear whether the failure generalizes to general text generation tasks; teacherless training works only in limited settings.
Experimental setup
Datasets: synthetic path-star graphs G_{d,l}(N)
Benchmarks: path-star graph tasks (G_{d,l}) 3-digit addition
Metrics: exact-match accuracy first-token accuracy clever hans cheat accuracy snowball accuracy
Baselines: standard teacher-forcing reversed-target training teacherless training
Builds on: teacher forcing Mamba GPT-2 teacherless training
Closest papers by meaning
- Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction (ICML 2025 · similarity 0.90)
- How reinforcement learning after next-token prediction facilitates learning (ICLR 2026 · similarity 0.89)
- Auto-Regressive Next-Token Predictors are Universal Learners (ICML 2024 · similarity 0.88)
- Rethinking Thinking Tokens: LLMs as Improvement Operators (ICML 2026 · similarity 0.87)
- On the Bias of Next-Token Predictors Toward Systematically Inefficient Reasoning: A Shortest-Path Case Study (NeurIPS 2025 · similarity 0.87)
- Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.