Research map / Large language models / Reasoning and chain of thought
Probing the Trajectories of Reasoning Traces in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a trajectory-probing protocol that truncates reasoning traces at token deciles, injects them back into LLMs, and measures next-token answer distributions, with length/form/token-identity controls and weak-to-strong rescue experiments on GPQA Diamond and MMLU-Pro.
Problem
It is unclear how accuracy and decision commitment evolve along an LLM reasoning trace, and whether intermediate trace segments carry answer-relevant information beyond length or stylistic effects.
Approach
Truncate each reasoning trace at token deciles, inject each prefix with an early-stopping suffix, and read next-token answer-letter probabilities to trace accuracy and commitment; compare against random, swap, and shuffle controls and test cross-model rescue versus anchoring.
Key results
- Accuracy rises from 30.8-42.3% at decile 0 to 49.0-73.1% at decile 100 on GPQA Diamond; MMLU-Pro from 40.4-50.4% to 61.1-79.4%.
- Random-token controls stay near baseline (-3.9% to +4.2%) while original traces gain +18.1-33.1%, indicating gains are driven by instance-specific content.
- Free continuation raises rescue rates at decile 20 on GPQA Diamond to 42.9-68.1% versus 20.8-34.1% in base mode, but anchoring remains substantial.
Surprising findings
- Qwen3-8B generates \boxed{} formatting in 27% of traces, biasing predictions toward 'A' and causing an accuracy drop from decile 90 to 100.
- Shorter reasoning traces are associated with higher accuracy, and confidence discrimination does not improve with reasoning depth.
Limitations
- Evaluates only multiple-choice benchmarks with single-letter answers, so open-ended generation is not covered.
- Does not cover closed-source frontier models.
- Does not address faithfulness of traces to internal computation; decile slicing treats positions uniformly.
Experimental setup
Datasets: GPQA Diamond MMLU-Pro
Benchmarks: GPQA Diamond MMLU-Pro
Metrics: accuracy decision commitment non-choice probability flip rate rescue rate anchoring rate McNemar's test
Baselines: random control swap control shuffle control no-reasoning baseline (decile 0)
Builds on: chain-of-thought test-time scaling McNemar's test
Closest papers by meaning
- Tracing the Traces: Latent Temporal Signals for Efficient and Accurate Reasoning (ICLR 2026 · similarity 0.93)
- Reasoning-Trace Collapse: Evaluating the Loss of Explicit Reasoning During Fine-Tuning (NeurIPS 2026 · similarity 0.90)
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (NeurIPS 2025 · similarity 0.90)
- When Confidence Rises Too Early: Detecting Shortcut Reasoning via Premature Answer Commitment (NeurIPS 2026 · similarity 0.90)
- Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens (NeurIPS 2026 · similarity 0.90)
- On the Bias of Next-Token Predictors Toward Systematically Inefficient Reasoning: A Shortest-Path Case Study (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.