AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Next-ToBE: Probabilistic Next Token-Bag Exploitation for Activating Anticipatory Capacity in LLMs

Yihe Liu, Huibin Wang, Xianming Hu, Pinyi Zhang, Jiahao Xiong, Chenglin Wang, Nuoyi Chen, Hongbo Zhao, Jie Zhang, Kai Zhang

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Next-ToBE, a method that profiles and enhances LLMs' latent capacity to anticipate long-range future tokens as a bag, and leverages it to improve reasoning.

Problem

Auto-regressive LLMs, trained to predict one next token at a time, show some capacity to anticipate long-range future tokens, but it is unclear how to systematically profile, enhance, and exploit this capacity to improve reasoning.

Approach

Treats the set of upcoming tokens as a probabilistic token bag and trains/exploits the model's anticipatory ability over that bag to activate it for reasoning gains. Details beyond the abstract are not available.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace one-hot next-token target with soft token-bag target (Next-ToBE)fine-tuned Qwen2.5-Math-1.5B · Math (5-benchmark avg)avg accuracy: 33.9 -> 38.3Improved
Next-ToBE versus MTP and MuToRQwen2.5-Math-7B and Llama3.1-8B-instruct · Math (5-benchmark avg)avg accuracy: 42.7 vs 40.3 / 41.1 (7B); 26.7 vs 23.3 / 24.5 (Llama)Improved
multi-token prediction heads (MTP) instead of NTPLlama3.1-8B-instruct · Math (5-benchmark avg)avg accuracy: 23.9 -> 23.3Hurt
Next-ToBE on code generationQwen2.5-1.5B/7B and Llama3.1-8B · Code generationavg pass@1: 52.6 -> 55.1 (1.5B); 71.6 -> 75.2 (7B); 64.5 -> 67.3 (Llama)Improved
Next-ToBE on commonsense reasoningQwen2.5 and Llama3.1 · Commonsense reasoningavg accuracy: 77.7 -> 79.5 (1.5B); 88.1 -> 89.4 (7B); 85.7 -> 87.8 (Llama)Improved
scale to larger modelNext-ToBE on Qwen3-14B · Math (5-benchmark avg)avg accuracy: 32.2 -> 34.9Improved
add normalization of look-ahead token probabilities (ablation)Next-ToBE · Math (5-benchmark avg)avg accuracy: 32.4 -> 38.3Improved
add random-walk multiplication fusion instead of direct multiplication (ablation)Next-ToBE weighting · Math (5-benchmark avg)avg accuracy: 37.4 -> 38.3Improved
use multiplication fusion instead of weighted summation (ablation)Next-ToBE weighting · Math (5-benchmark avg)avg accuracy: 37.8 -> 38.3Improved
tighten or loosen probability threshold epsilon away from 1e-8 (ablation)Next-ToBE · Math (5-benchmark avg)avg accuracy: 38.3 -> 36.6 (1e-4), 36.4 (1e-12)Hurt
increase lambda beyond optimum (ablation)Next-ToBE · Math and codeaccuracyMixed
Next-ToBE in pretraining from scratch versus NTPGPT-2 124M · WikiText-103FtHR (top-50, k=10) / perplexity: FtHR 22.51 -> 25.80; PPL 28.67 -> 34.32Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy