Research map / Architectures and efficiency / Attention and transformers
Predicting the Order of Upcoming Tokens Improves Language Modeling
Zayd Muhammad Kawakibi Zuhri, Erland Hilman Fuadi, Alham Fikri Aji
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TOP auxiliary loss to NTP | 340M-7B LMs · 9 standard NLP benchmarks | accuracy / perplexity: 7B Lambada acc 55.89 -> 57.03 | Improved |
| add MTP auxiliary heads to NTP | 7B LM · standard NLP benchmarks | accuracy: 7B Lambada acc 55.89 -> 53.13 | Hurt |
| use DS-MTP instead of MTP | LMs · standard NLP benchmarks | accuracy | No clear effect |
| add TOP loss then continue training on math | 1.8B LM · GSM8K | accuracy (%): 39.20 -> 45.64 | Improved |
| add TOP loss then continue training on math | 7B LM · MATH | accuracy (%): 21.74 -> 20.40 | Hurt |
| add DS-MTP then continue training on math/code | 1.8B LM · GSM8K | accuracy (%): 39.20 -> 2.65 | Hurt |
| add TOP loss on code continued training | 7B LM · MBPP pass@64 | accuracy (%): 49.20 -> 54.00 | Improved |
| add TOP loss on star graph pathfinding | 8-layer transformer · star graph G(5,5) | test accuracy (%): 0.1 -> 100 | Improved |
| use more MTP heads (4 instead of 2) (ablation) | MTP · star graph G(3,5) | test accuracy (%): 59.0 -> 100 | Improved |
| use TOP head for self-speculative decoding vs MTP heads | TOP · self-speculative decoding | accepted tokens per pass: 7B math: MTP 2.49, TOP 1.55 | Hurt |
| vary TOP window size from 4 to 4096 (ablation) | TOP 340M · Lambada accuracy | accuracy (%): 36.35 (NTP) -> 36.95-38.68 | Improved |
| raise TOP loss weight alpha toward 0.9 (ablation) | TOP 340M · Lambada accuracy | accuracy (%): 37.07 (alpha 0.5) -> 38.39 (alpha 0.9) | Improved |
Closest papers by meaning
- Better & Faster Large Language Models via Multi-token Prediction (ICML 2024 · similarity 0.90)
- L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models (NeurIPS 2025 · similarity 0.90)
- On the Generalization Ability of Next-Token-Prediction Pretraining (ICML 2025 · similarity 0.87)
- Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing (ICML 2026 · similarity 0.87)
- Beyond Next Token Prediction: Patch-Level Training for Large Language Models (ICLR 2025 · similarity 0.86)
- Reinforced Fast Weights via Next-Sequence Prediction (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
