Research map / Large language models / Reasoning and chain of thought

Attention Illuminates LLM Reasoning: The Uncovered Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses attention-derived metrics (WAAD and FAI) to identify preplan and anchor tokens in LLM reasoning, then reweights GRPO token advantages toward these tokens for fine-grained credit assignment.

Problem

RL for LLM reasoning assigns sequence-level rewards uniformly across all tokens, blurring pivotal reasoning steps from routine ones, and the internal reasoning structure of LLMs remains opaque.

Approach

Classify attention heads as local or global by attention span, define WAAD (local lookback within a clipped window) and FAI (average attention received from future tokens) from an auxiliary forward pass, and use these to identify preplan and anchor tokens; amplify their advantages in GRPO, with a coupled variant that back-allocates credit from locally dominated anchors to their introductory tokens

Key results

• On Qwen3-8B-Base (1K length), coupled rhythm credit raises the math average from 39.4 (GRPO) to 43.2, with +5.0 on AIME25 and +6.3 on AMC23.
• On Countdown, coupled rhythm credit reaches 63.1% versus 52.6% for GRPO on Qwen3-4B-Base.
• On Qwen3-4B-Base at 8K length, coupled rhythm credit improves the math average from 44.8 to 47.3.

Surprising findings

• Reinforcing low-scoring (bottom-40%) tokens by the same metrics degrades performance below GRPO, e.g., the coupled bottom-40% variant averages 32.5 vs 39.4.
• Random and high-entropy token selection provide marginal or no gains over GRPO.

Limitations

• Attention maps are obtained from an auxiliary forward pass on a standard Transformer rather than the training model, and the authors state attention is not an unbiased or complete causal explanation.
• Gains on CrossThink-QA are smaller than on structured tasks, and the rhythm is weaker in QA settings.
• Hyperparameters (top-40% quantile, gamma_amp=1.5, layer range) are chosen with ablations on evaluation benchmarks, which may bias reported gains.

Experimental setup

Datasets: DAPO-Math-17K Countdown CrossThink-QA GSM8K

Benchmarks: AIME24 AIME25 AMC23 MATH500 OlympiadBench Countdown CrossThink-QA Minerva GSM8K ZebraLogic

Metrics: accuracy avg@16 pass@1

Baselines: GRPO random credit high-entropy credit high-entropy selection CAPO ThinkPRM-1.5B AsyPPO

Builds on: GRPO PPO RLVR ROLL

Closest papers by meaning

• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.90)
• How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMs (ICML 2026 · similarity 0.90)
• Don't Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models (ICLR 2025 · similarity 0.89)
• Revealing Behavioral Plasticity in Large Language Models: A Token-Conditional Perspective (ICML 2026 · similarity 0.89)
• The Tell-Tale Norm: $\ell_2$ Magnitude as a Signal for Reasoning Dynamics in Large Language Models (ICML 2026 · similarity 0.89)
• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.89)