Research map / Large language models / Reasoning and chain of thought

Towards Long-Horizon Interpretability: Efficient and Faithful Multi-Token Attribution for Reasoning LLMs

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FlashTrace, a multi-token attribution method for reasoning LLMs that aggregates attribution over a target span in one pass and recursively traces importance through reasoning tokens back to inputs, reporting large speedups and better faithfulness.

Problem

Token attribution for LLM outputs scales poorly to multi-token targets (O(M·N) cost) and attribution mass gets absorbed by intermediate reasoning tokens, so importance does not propagate back to input tokens.

Approach

Span-wise aggregation of attention weights and residual/MLP contributions over a multi-token target so attribution costs one pass per hop, plus a recursive hop that re-attributes the weighted reasoning span back to the input and combines hops with flow-ratio discounting.

Key results

• FlashTrace is over 130x faster than the most efficient baseline (IFR) at a 5k-token target span, under 20 seconds versus over 38 minutes.
• On RULER HotpotQA (1024 setting), FlashTrace reaches RISE 0.033 versus 0.074 for IFR (lower is better).
• On MATH and MorehopQA, FlashTrace achieves the lowest RISE and MAS among compared methods, e.g., MorehopQA RISE 0.128 vs 0.146 for IFR.

Surprising findings

• Naive attribution's attribution mass on reasoning tokens grows to over 90% as reasoning lengthens, reducing recovery of ground-truth input tokens.
• On the MATH task, a single exhaustive token-level rollout is slightly more faithful than FlashTrace (RISE 0.116 vs 0.128), yet FlashTrace is about 94% faster.

Limitations

• Proximity scores measure informational contribution, not direct causal effect; faithfulness is validated only via perturbation metrics.
• Attention maps across all layers require O(L·N^2) memory if cached, and linearization of normalization is an approximation inherited from ALTI/IFR.
• Integrated Gradients and IG-Attn ran out of memory on most RULER samples, so their faithfulness could not be measured.

Experimental setup

Datasets: RULER HotpotQA MATH MoreHopQA Aider

Benchmarks: RULER (Needle-in-a-Haystack, Variable Tracking) HotpotQA MATH MoreHopQA Aider

Metrics: Recovery Rate RISE MAS wall-clock time peak GPU memory

Baselines: Integrated Gradients IG-Attn Perturbation (LOO) REAGENT CLP IFR AttnLRP CAGE

Builds on: ALTI IFR Integrated Gradients AttnLRP

Closest papers by meaning

• Segment-Level Attribution for Selective Learning of Long Reasoning Traces (ICLR 2026 · similarity 0.89)
• Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (ICML 2025 · similarity 0.89)
• On the Token Value Inequality in Efficient Reasoning (NeurIPS 2026 · similarity 0.89)
• On the Bias of Next-Token Predictors Toward Systematically Inefficient Reasoning: A Shortest-Path Case Study (NeurIPS 2025 · similarity 0.89)
• Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens (NeurIPS 2026 · similarity 0.88)
• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.88)