Research map / Large language models / Language, safety and interpretability
How do Transformers Learn Implicit Reasoning?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains GPT-2-style transformers from scratch on a controlled symbolic two-hop reasoning environment to trace how implicit multi-hop reasoning emerges, and introduces cross-query semantic patching and a cosine-based representational lens to link reasoning to clustering of intermediate entity representations.
Problem
How LLMs acquire and perform implicit multi-hop reasoning during training, i.e., whether models genuinely compose facts internally or merely memorize answers, and what internal mechanisms support it.
Approach
Use a controlled symbolic environment with ID/OOD triple partitions and query-level ablations to observe a three-stage developmental trajectory, then explain behavior via cross-query semantic patching (localizing bridge entities at r1 tokens) and cosine-space clustering of intermediate representations rather than logit-lens decoding.
Key results
- Training shows three phases: memorization, ID generalization (Test-II), then cross-distribution generalization (Test-OI) that arrives later; OOD second-hop queries never generalize.
- Explicit decodability of intermediate entities is high even in Phase I (ID-derived 92.1% immediate probing) and does not track reasoning emergence, while ID cohesion in cosine space rises with Test-II accuracy.
- Training with only Train-II queries (no atomic triples) still yields ID generalization, with atomic ID triples accelerating it; second-hop generalization fails unless the exact composition appears in training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with Train-II queries only, without atomic triples (ablation) | GPT-2 · Test-II | accuracy | Helped |
| Add atomic ID triples to Train-II query training (ablation) | GPT-2 · Test-II | time to generalization | Helped |
| Remove selected triples from second-hop positions in training (ablation) | GPT-2 · Test-II queries using those triples as second hops | accuracy | Hurt |
| Increase second-hop triple exposure frequency during training (ablation) | GPT-2 · Test-II at early Phase II | accuracy | Helped |
| Use an ID/OOD atomic-triple ratio of 0.8/0.2 rather than 0.5/0.5 or 0.3/0.7 (ablation) | GPT-2 base configuration · Test-OI | accuracy | Helped |
| Remove all ID triples while retaining OOD triples and Train-II queries (ablation) | GPT-2 · Test-OI | accuracy | Hurt |
| Remove ID triples while retaining Train-II queries (ablation) | GPT-2 · Test-II | time to generalization | Hurt |
| Encourage explicit decoding of intermediate entities with OOD triple supervision (ablation) | GPT-2 · Test-II | accuracy | Mixed |
| Remove exposure to atomic triples as second hops, while retaining their other training roles (ablation) | GPT-2 base configuration · Test-II queries requiring the held-out second-hop triples | accuracy | Hurt |
| Increase ID-derived and OOD-derived representational alignment across training phases | GPT-2 · Test-OI | accuracy | Helped |
| Increase ID-derived representational cohesion across training | GPT-2 · Test-II | accuracy | Helped |
| Use larger Qwen2.5-1.5B rather than the smaller model | Qwen2.5-1.5B · Test-II | accuracy relative to ID Cohesion | Mixed |
Surprising findings
- Intermediate entities are explicitly decodable from hidden states yet decodability does not predict when reasoning emerges.
- First-hop OOD generalization in Test-OI appears to be an artifact of representational alignment induced by ID triples rather than true generalization.
Limitations
- Controlled symbolic setting with a small vocabulary (2000 entities, 200 relations) may not reflect LLM pretraining knowledge.
- Larger-model (Qwen2.5-1.5B) experiment shows high variance and decoupling of Test-II from cohesion, only partially validating findings.
Experimental setup
Datasets: synthetic symbolic 2-hop and 3-hop compositional queries built from atomic triples
Benchmarks: Test-II Test-OI Test-IO Test-OO (synthetic symbolic test sets) Test-III/OII (3-hop)
Metrics: accuracy patching success rate decoding success rate ID cohesion score OOD alignment score
Baselines: logit lens probing linear probing activation patching
Builds on: grokking logit lens activation patching
Closest papers by meaning
- Grokking of Implicit Reasoning in Transformers: A Mechanistic Journey to the Edge of Generalization (NeurIPS 2024 · similarity 0.92)
- SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.89)
- Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models (ICML 2026 · similarity 0.89)
- Multi-head Transformers Provably Learn Symbolic Multi-step Reasoning via Gradient Descent (NeurIPS 2025 · similarity 0.89)
- Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data? (ICLR 2025 · similarity 0.89)
- Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.