Research map / Large language models / Reasoning and chain of thought

How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMs

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FlowTracer, which builds an answer-conditioned attention DAG, reweights it with a Doob-h transform for flow conservation, and uses per-token flow throughput to upweight the top 40% tokens in GRPO for RLVR credit assignment.

Problem

Token-level credit assignment in RLVR for LLMs: standard GRPO weights all tokens uniformly, and existing internal-signal methods (entropy, attention) are point-wise and ignore how information propagates globally to the answer.

Approach

Model attention as an influence-coupling DAG, condition it on the answer region via a Doob-h-like transform (ensuring local flow conservation and removing dead-end branches), propagate unit flow from question to answer, and use token throughput to scale GRPO token weights.

Key results

• On Qwen3-8B (1K context), FlowTracer averages 43.4% vs. GRPO 39.4% across five math benchmarks (+4.0).
• On Qwen3-4B at 8K context, average rises from 44.8% (GRPO) to 48.6%, with +5.8 on AIME25.
• Perturbing 20% high-flow tokens changes the answer in 45.9% of cases vs. 29.5% for random and 14.9% for low-flow on GSM8K; computational overhead is 2.1-4.5%.

Surprising findings

• Structural delimiters alone recover most of the gain (38.8% vs. 39.4% full), with semantic-content tokens adding a smaller improvement (37.6%).
• Continuous flow-based weighting (raw flow, sigmoid, log1p) underperforms a hard Top-40% mask, indicating flow works better as a ranking signal.

Limitations

• Assumes a localized answer region; extending to open-ended generation or tool-call trajectories needs different target definitions.
• Outcome-only rewards cannot separate useful intermediate reasoning from reasoning supporting wrong answers.
• Very long contexts may yield noisier attention graphs.

Experimental setup

Datasets: DAPO-Math-17K Countdown (TinyZero protocol) CrossThinkQA GSM8K

Benchmarks: AIME24 AIME25 AMC23 MATH500 OlympiadBench CrossThinkQA Countdown Minerva GSM8K

Metrics: accuracy (avg pass rate %) answer change rate correctness reverse rate time overhead

Baselines: GRPO Random High-entropy selection Gradient Correlation Attention CAPO ThinkPRM-1.5B AsyPPO Reweight+Lopti

Builds on: GRPO GAE Doob h-transform

Closest papers by meaning

• Attention Illuminates LLM Reasoning: The Uncovered Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization (ICML 2026 · similarity 0.90)
• On the Bias of Next-Token Predictors Toward Systematically Inefficient Reasoning: A Shortest-Path Case Study (NeurIPS 2025 · similarity 0.88)
• Probing the Trajectories of Reasoning Traces in Large Language Models (NeurIPS 2026 · similarity 0.88)
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.88)
• Tracing the Traces: Latent Temporal Signals for Efficient and Accurate Reasoning (ICLR 2026 · similarity 0.88)
• SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.88)