Research map / Large language models / Reasoning and chain of thought

How Does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses sparse autoencoders and activation patching on Pythia-70M and Pythia-2.8B over GSM8K to test whether chain-of-thought prompting induces causally effective, sparser, more interpretable features, finding effects only in the larger model.

Problem

Whether chain-of-thought reasoning reflects the model's true internal computation, and whether CoT prompting changes internal features in a causally meaningful, faithful way, is unresolved at the feature level for math word problems.

Approach

Combines SAEs trained separately on CoT and noCoT residual activations with feature-level activation patching (Top-K and Random-K, patch curves) and log-probability evaluation, plus sparsity and auto-interpretation score comparisons, to test CoT faithfulness mechanistically.

Key results

• In Pythia-2.8B, patching top-K CoT features into noCoT runs raises correct-answer log-probability (CoT-to-noCoT curve exceeds +3.2 at K=2 at dictionary ratio 8), while Pythia-70M shows no gain or declines.
• Under CoT, Pythia-2.8B feature explanation scores average 0.056 vs -0.013 for noCoT (t=2.96, p=0.004); Pythia-70M shows no significant difference (0.018 vs 0.016, p=0.935).
• In Pythia-2.8B, randomly sampled CoT features often outperform Top-K features when patched into noCoT runs, suggesting the useful signal is distributed.

Surprising findings

• Randomly selected CoT features often outperform top-activated features in the 2.8B model.
• Pythia-70M shows a fragile benefit from NoCoT-to-CoT patching at small K, and CoT-to-noCoT patching degrades performance.

Limitations

• Activation patching targets only the final-token residual stream at layer 2 and does not trace causal effects through the reasoning process.
• Only Pythia-70M and Pythia-2.8B are tested; results may not generalize to larger models.
• Interpretation relies on GPT-3.5-turbo scoring without causal validation of explanations.

Experimental setup

Datasets: GSM8K

Benchmarks: GSM8K

Metrics: change in log-probability of correct answer explanation score activation sparsity t-test p-value

Baselines: Random-K patching

Builds on: sparse autoencoders activation patching zero-shot chain-of-thought automated interpretability

Closest papers by meaning

• Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns (AAAI 2025 · similarity 0.92)
• Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysis (ICLR 2025 · similarity 0.91)
• Chain-of-Thought Reasoning Without Prompting (NeurIPS 2024 · similarity 0.91)
• Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and Beyond (ICLR 2026 · similarity 0.91)
• From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency (ICLR 2025 · similarity 0.91)
• To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (ICLR 2025 · similarity 0.91)