Research map / Large language models / Language, safety and interpretability
Who Reasons in the Large Language Models?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Stethoscope for Networks (SfN), a set of weight-diff, module-merge, freeze-tuning, and module-destruction diagnostics, to argue that the attention output projection (o_proj) is primarily responsible for reasoning in LLMs while other modules support conversation.
Problem
Whether reasoning ability in LLMs stems from the whole model, specific modules, or is an artifact of overfitting, given that the mechanism behind SFT/RL-induced reasoning is opaque.
Approach
Uses diagnostic tools that compare base and reasoning-distilled model weights (Delta Stethoscope), swap modules between them without tuning (Merge Stethoscope), fine-tune only selected modules (Freeze Stethoscope), and destroy modules (Destruction Stethoscope) to localize reasoning to o_proj.
Key results
- Tuning only o_proj and layernorm (plus embeddings and lm_head) on Qwen2.5-32B-Instruct reaches AIME 2024 0.367, matching the full-tuned model (0.367) and exceeding base (0.167), with 3.2B vs 32.8B trainable parameters.
- For Qwen2.5-14B, o_proj-only tuning reaches AIME 2024 0.266, equal to full tuning, and Math 500 0.848 vs 0.872 for full tuning.
- Merging only o_proj from DeepSeek-R1-Distill-Qwen-1.5B into Qwen2.5-Math-1.5B, without tuning, yields correct reasoning on some AIME 2024 questions the base fails on.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add tuning of o_proj to Emb + Head-only tuning (ablation) | Qwen2.5-32B-Instruct (Freeze Stethoscope) · AIME 2024 | accuracy: 0.200 -> 0.367 | Helped |
| Add tuning of o_proj to Emb + Head-only tuning (ablation) | Qwen2.5-32B-Instruct (Freeze Stethoscope) · Math 500 | accuracy: 0.756 -> 0.890 | Helped |
| Add tuning of o_proj to Emb + Head-only tuning (ablation) | Qwen2.5-32B-Instruct (Freeze Stethoscope) · GPQA Diamond | accuracy: 0.444 -> 0.520 | Helped |
| Add tuning of o_proj to Emb + Head-only tuning (ablation) | Qwen2.5-14B-Instruct (Freeze Stethoscope) · AIME 2024 | accuracy: 0.133 -> 0.266 | Helped |
| Add tuning of o_proj to Emb + Head-only tuning (ablation) | Qwen2.5-14B-Instruct (Freeze Stethoscope) · Math 500 | accuracy: 0.722 -> 0.848 | Helped |
| Add tuning of o_proj to Emb + Head-only tuning (ablation) | Qwen2.5-14B-Instruct (Freeze Stethoscope) · GPQA Diamond | accuracy: 0.414 -> 0.485 | Helped |
| Also tune q_proj, k_proj, and v_proj (ablation) | Qwen2.5-32B-Instruct with Emb + Head + o_proj tuning · AIME 2024 | accuracy: 0.367 -> 0.300 | Hurt |
| Also tune q_proj, k_proj, and v_proj (ablation) | Qwen2.5-32B-Instruct with Emb + Head + o_proj tuning · GPQA Diamond | accuracy: 0.520 -> 0.525 | Mixed |
| Also tune q_proj, k_proj, and v_proj (ablation) | Qwen2.5-14B-Instruct with Emb + Head + o_proj tuning · AIME 2024 | accuracy: 0.266 -> 0.233 | Hurt |
| Also tune q_proj, k_proj, and v_proj (ablation) | Qwen2.5-14B-Instruct with Emb + Head + o_proj tuning · Math 500 | accuracy: 0.848 -> 0.854 | Mixed |
| Restrict tuning to o_proj (with Emb + Head) instead of tuning all parameters (ablation) | Qwen2.5-32B-Instruct · Math 500 | accuracy: 0.906 -> 0.890 | Hurt |
| Restrict tuning to o_proj (with Emb + Head) instead of tuning all parameters (ablation) | Qwen2.5-14B-Instruct · AIME 2024 | accuracy: 0.266 -> 0.266 | No clear effect |
Surprising findings
- Replacing only o_proj weights from a reasoning model into a base model, without any further training, produces coherent reasoning outputs.
- Tuning only o_proj plus normalization runs about 3x faster than full fine-tuning in the reported setup, and full unfreezing shows overfitting signs in loss curves.
Limitations
- Findings are based on a limited set of model families and reasoning benchmarks and may not generalize.
- Several diagnostic results rely on qualitative assessment rather than statistical validation; destruction results come from a single conversation.
- Lacks a theoretical account of why o_proj drives reasoning; the Freeze experiments deviate from the s1 testing recipe and use no hyperparameter tuning.
Experimental setup
Datasets: s1K AIME 2024 MATH 500 GPQA Diamond
Benchmarks: AIME 2024 MATH 500 GPQA Diamond
Metrics: accuracy training steps/s trainable parameter count
Baselines: s1 (full fine-tuning) Qwen2.5-32B-Instruct Qwen2.5-14B-Instruct
Builds on: s1 DeepSeek-R1-Distill-Qwen Qwen2.5
Closest papers by meaning
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.90)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.90)
- Improving Reasoning Performance in Large Language Models via Representation Engineering (ICLR 2025 · similarity 0.89)
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.89)
- Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models (ICLR 2025 · similarity 0.89)
- Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.