Research map / Large language models / Reasoning and chain of thought
Next-ToBE: Probabilistic Next Token-Bag Exploitation for Activating Anticipatory Capacity in LLMs
Yihe Liu, Huibin Wang, Xianming Hu, Pinyi Zhang, Jiahao Xiong, Chenglin Wang, Nuoyi Chen, Hongbo Zhao, Jie Zhang, Kai Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Next-ToBE, a method that profiles and enhances LLMs' latent capacity to anticipate long-range future tokens as a bag, and leverages it to improve reasoning.
Problem
Auto-regressive LLMs, trained to predict one next token at a time, show some capacity to anticipate long-range future tokens, but it is unclear how to systematically profile, enhance, and exploit this capacity to improve reasoning.
Approach
Treats the set of upcoming tokens as a probabilistic token bag and trains/exploits the model's anticipatory ability over that bag to activate it for reasoning gains. Details beyond the abstract are not available.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace one-hot next-token target with soft token-bag target (Next-ToBE) | fine-tuned Qwen2.5-Math-1.5B · Math (5-benchmark avg) | avg accuracy: 33.9 -> 38.3 | Improved |
| Next-ToBE versus MTP and MuToR | Qwen2.5-Math-7B and Llama3.1-8B-instruct · Math (5-benchmark avg) | avg accuracy: 42.7 vs 40.3 / 41.1 (7B); 26.7 vs 23.3 / 24.5 (Llama) | Improved |
| multi-token prediction heads (MTP) instead of NTP | Llama3.1-8B-instruct · Math (5-benchmark avg) | avg accuracy: 23.9 -> 23.3 | Hurt |
| Next-ToBE on code generation | Qwen2.5-1.5B/7B and Llama3.1-8B · Code generation | avg pass@1: 52.6 -> 55.1 (1.5B); 71.6 -> 75.2 (7B); 64.5 -> 67.3 (Llama) | Improved |
| Next-ToBE on commonsense reasoning | Qwen2.5 and Llama3.1 · Commonsense reasoning | avg accuracy: 77.7 -> 79.5 (1.5B); 88.1 -> 89.4 (7B); 85.7 -> 87.8 (Llama) | Improved |
| scale to larger model | Next-ToBE on Qwen3-14B · Math (5-benchmark avg) | avg accuracy: 32.2 -> 34.9 | Improved |
| add normalization of look-ahead token probabilities (ablation) | Next-ToBE · Math (5-benchmark avg) | avg accuracy: 32.4 -> 38.3 | Improved |
| add random-walk multiplication fusion instead of direct multiplication (ablation) | Next-ToBE weighting · Math (5-benchmark avg) | avg accuracy: 37.4 -> 38.3 | Improved |
| use multiplication fusion instead of weighted summation (ablation) | Next-ToBE weighting · Math (5-benchmark avg) | avg accuracy: 37.8 -> 38.3 | Improved |
| tighten or loosen probability threshold epsilon away from 1e-8 (ablation) | Next-ToBE · Math (5-benchmark avg) | avg accuracy: 38.3 -> 36.6 (1e-4), 36.4 (1e-12) | Hurt |
| increase lambda beyond optimum (ablation) | Next-ToBE · Math and code | accuracy | Mixed |
| Next-ToBE in pretraining from scratch versus NTP | GPT-2 124M · WikiText-103 | FtHR (top-50, k=10) / perplexity: FtHR 22.51 -> 25.80; PPL 28.67 -> 34.32 | Mixed |
Closest papers by meaning
- Reinforced Fast Weights via Next-Sequence Prediction (NeurIPS 2026 · similarity 0.90)
- Better & Faster Large Language Models via Multi-token Prediction (ICML 2024 · similarity 0.90)
- Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.89)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.89)
- L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models (NeurIPS 2025 · similarity 0.89)
- Skip a Layer or Loop It? Learning Program-of-Layers in LLMs (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
