Research map / Architectures and efficiency / RNNs and normalization
Feature Resemblance: Towards a Theoretical Understanding of Analogical Reasoning in Transformers
Ruichen Xu, Wenjing Yan, Ying-Jun Angela Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn similarity premises before attribution premises (curriculum) instead of reverse (ablation) | GPT-2 trained from scratch on synthetic triples · Synthetic analogical reasoning | success rate: 0.7% -> 100% | Improved |
| learn similarity premises before attribution premises (ablation) | One-layer transformer · Synthetic analogical reasoning | success rate: 0% -> 100% | Improved |
| joint training on similarity and attribution premises | GPT-2 · Synthetic analogical reasoning | feature similarity: 0.995±6e-4 with 100% success | Improved |
| add identity bridge examples (b -> b) to two-hop training data (ablation) | GPT-2 · Synthetic two-hop reasoning | success rate: 0% -> 99.7% | Improved |
| add identity bridge examples (ablation) | One-layer transformer · Synthetic two-hop reasoning | success rate: 1.7% -> 100% | Improved |
| add identity bridge examples (ablation) | GPT-2 · Synthetic two-hop reasoning | feature similarity: 0.016 -> 0.9642 | Improved |
| learn similarity before attribution (late on attribution vs late on similarity) (ablation) | Llama-3-1B · Natural-language analogy | success rate: 53% -> 86% | Improved |
| learn similarity before attribution (ablation) | Qwen-2.5-1.5B · Natural-language analogy | success rate: 33% -> 72% | Improved |
| learn similarity before attribution (ablation) | Qwen-3-8B · Natural-language analogy | success rate: 34% -> 75% | Improved |
| joint training instead of late-on-attribution (ablation) | Llama-3-1B · Natural-language analogy | success rate: 86% -> 73% | Hurt |
| compare joint vs late-on-attribution (ablation) | Qwen-2.5-1.5B and Qwen-3-8B · Natural-language analogy | success rate: 85% vs 72% (Qwen-2.5-1.5B); 84% vs 75% (Qwen-3-8B) | No clear effect |
Closest papers by meaning
- Grokking of Implicit Reasoning in Transformers: A Mechanistic Journey to the Edge of Generalization (NeurIPS 2024 · similarity 0.89)
- When can transformers reason with abstract symbols? (ICLR 2024 · similarity 0.89)
- Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data? (ICLR 2025 · similarity 0.89)
- How do Transformers Learn Implicit Reasoning? (NeurIPS 2025 · similarity 0.88)
- Latent Concept Disentanglement in Transformer-based Language Models (ICLR 2026 · similarity 0.88)
- Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
