Research map / Large language models / Language, safety and interpretability
Emergent Analogical Reasoning in Transformers
Gouki Minegishi, Jingyuan Feng, Hiroki Furuta, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reduce number of relations to 100 (ablation) | 1-layer Transformer · Synthetic analogy task, 1-layer Transformer | OOD analogical accuracy: analogical reasoning fails at |R| = 100 | Hurt |
| increase number of entities (ablation) | 1-layer Transformer · Synthetic analogy task, 1-layer Transformer | time to acquire analogical reasoning | Hurt |
| raise analogical OOD ratio to 0.9 (ablation) | 1-layer Transformer · Synthetic analogy task, 1-layer Transformer | OOD analogical accuracy | Hurt |
| raise compositional OOD ratio (ablation) | 1-layer Transformer · Synthetic analogy task, 1-layer Transformer | analogical reasoning | No clear effect |
| raise weight decay from 0 to 0.01 or 0.1 (ablation) | 1-layer Transformer · Synthetic analogy task, 1-layer Transformer | emergence time of analogical reasoning | Improved |
| raise weight decay to 1.0 (ablation) | 1-layer Transformer · Synthetic analogy task, 1-layer Transformer | OOD analogical accuracy: fails to be learned; compositional reasoning remains robust | Hurt |
| increase batch size (ablation) | 1-layer Transformer · Synthetic analogy task, 1-layer Transformer | emergence time of analogical reasoning | Improved |
| increase width from 64 to 128 or 256 (ablation) | Transformer · Synthetic analogy task, 1-layer Transformer | OOD analogical accuracy | Improved |
| increase width to 512 (ablation) | Transformer · Synthetic analogy task, 1-layer Transformer | OOD analogical accuracy | Hurt |
| increase model size (ablation) | Transformer · Synthetic analogy task, 1-layer Transformer | compositional reasoning accuracy | Improved |
| increase depth to 2 or 4 layers (fixed default optimization) (ablation) | Transformer · Synthetic analogy task, 1-layer Transformer | OOD analogical accuracy | Hurt |
| add noise to functor-transferred relations (imperfect isomorphism) (ablation) | 1-layer Transformer · Synthetic analogy task, 1-layer Transformer | OOD analogical accuracy | Hurt |
Closest papers by meaning
- When can transformers reason with abstract symbols? (ICLR 2024 · similarity 0.89)
- Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data? (ICLR 2025 · similarity 0.88)
- Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models (ICML 2025 · similarity 0.88)
- Grokking of Implicit Reasoning in Transformers: A Mechanistic Journey to the Edge of Generalization (NeurIPS 2024 · similarity 0.88)
- Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalization (NeurIPS 2025 · similarity 0.87)
- Latent Concept Disentanglement in Transformer-based Language Models (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
