Research map / Large language models / Reasoning and chain of thought
L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes leap multi-token prediction (L-MTP), which assigns extra LM heads to non-adjacent (strided) future positions and uses a look-backward decoding with tree attention to speed up self-speculative inference, evaluated by fine-tuning Qwen2.5, Llama 3.x, and Gemma 3 models.
Problem
Next-token prediction is sequential, limiting contextual coverage and inference efficiency; standard multi-token prediction (MTP) only predicts adjacent future tokens.
Approach
Reassign the multiple prediction heads to leaping positions (stride k, e.g., positions 1,3,5,7) instead of adjacent positions, and decode by reusing predictions from prior steps (looking backward) so that the skipped tokens are filled in, combined with tree-attention verification for self-speculative decoding.
Key results
- L-MTP average score 36.80 vs NTP 36.40 and MTP 35.87 on Llama3.1-8B; 49.58 vs NTP 46.87 and MTP 45.17 on Gemma3-12B.
- On Gemma3-12B, L-MTP improves GSM8K to 26.38 vs 13.42 (NTP) and 5.61 (MTP).
- L-MTP extended to Medusa on Vicuna 13B reaches 2.43x speedup on GSM8K vs 2.24x for MTP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use L-MTP instead of MTP | Llama3.2-3B · Avg. across reported benchmarks | average score: 25.46 -> 26.68 | Helped |
| Use L-MTP instead of MTP | Llama3.1-8B · Avg. across reported benchmarks | average score: 35.87 -> 36.80 | Helped |
| Use L-MTP instead of MTP | Qwen2.5-3B · Avg. across reported benchmarks | average score: 52.79 -> 53.75 | Helped |
| Use L-MTP instead of MTP | Qwen2.5-7B · Avg. across reported benchmarks | average score: 63.34 -> 64.16 | Helped |
| Use L-MTP instead of MTP | Gemma3-4B · Avg. across reported benchmarks | average score: 36.59 -> 37.01 | Helped |
| Use L-MTP instead of MTP | Gemma3-12B · Avg. across reported benchmarks | average score: 45.17 -> 49.58 | Helped |
| Use L-MTP decoding instead of MTP decoding | Medusa with Vicuna 7B · GSM8K | speedup ratio: 1.83$\times$ -> 2.32$\times$ | Helped |
| Use L-MTP decoding instead of MTP decoding | Medusa with Vicuna 7B · MBPP | speedup ratio: 1.97$\times$ -> 2.01$\times$ | Helped |
| Use L-MTP decoding instead of MTP decoding | Medusa with Vicuna 13B · GSM8K | speedup ratio: 2.24$\times$ -> 2.43$\times$ | Helped |
| Use L-MTP decoding instead of MTP decoding | Medusa with Vicuna 13B · MBPP | speedup ratio: 1.98$\times$ -> 2.02$\times$ | Helped |
| Increase MTP head count from n=4 to n=7 (ablation) | Qwen2.5-3B with MTP · Avg. across reported benchmarks | average score: 52.79 -> 51.82 | Hurt |
| Set leap interval k=3 and heads n=3 instead of k=2 and n=4 (ablation) | Qwen2.5-3B with L-MTP · Avg. across reported benchmarks | average score: 53.75 -> 52.35 | Hurt |
Surprising findings
- On Qwen2.5-7B and Gemma3-4B, L-MTP scores below NTP on Math500 and GSM8K for some settings, and NTP itself sometimes degrades below base models.
- Simply increasing MTP horizon to n=7 lowered the average score (51.82) versus n=4 (52.79), while L-MTP with k=3, n=3 reached 52.35.
Limitations
- Experiments limited to models up to 12B parameters due to compute constraints.
- Gains are small and inconsistent across tasks; some tasks show regressions relative to NTP or MTP.
- Choice of n and k is fixed heuristically (k=2, n=4); adaptive selection left to future work.
Experimental setup
Datasets: MATH training set Evol-Instruct-Code (Magicoder) Alpaca-GPT4
Benchmarks: MATH500 GSM8K MBPP MBPP+ HumanEval HumanEval+ MMLU IFEval
Metrics: accuracy pass@1 speedup ratio
Baselines: NTP MTP F-MTP Medusa
Builds on: multi-token prediction Medusa speculative decoding self-distillation
Closest papers by meaning
- Better & Faster Large Language Models via Multi-token Prediction (ICML 2024 · similarity 0.92)
- Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing (ICML 2026 · similarity 0.90)
- Beyond Next Token Prediction: Patch-Level Training for Large Language Models (ICLR 2025 · similarity 0.89)
- Optimized Multi-Token Joint Decoding With Auxiliary Model for LLM Inference (ICLR 2025 · similarity 0.88)
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (ICML 2024 · similarity 0.88)
- Towards Auto-Regressive Next-Token Prediction: In-context Learning Emerges from Generalization (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.