Research map / Architectures and efficiency / Attention and transformers
Exploring Depth Generalization in Large Language Models for Solving Recursive Logic Tasks
Zhiyuan He
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use looped locate-and-replace instead of end-to-end | GPT-2 · Boolean algebra, depth 12 | accuracy: 51.8% -> 66.7% | Improved |
| use looped locate-and-replace instead of end-to-end | GPT-2 · Compositional arithmetic and propositional logic | accuracy | Improved |
| remove locate-and-replace from looped pipeline (Looped Only) (ablation) | LLR · Boolean algebra | accuracy | Improved |
| increase model size from 4 layers to 8 layers (ablation) | GPT-2 · Boolean algebra depth up to 10 | accuracy: difference never exceeds 1% | No clear effect |
| use ALiBi instead of NoPE/absolute/RoPE for locator (ablation) | LLR locator · length 20 generalization | accuracy: ALiBi 96% | Improved |
| increase recursion depth vs increase length | End-to-end GPT-2 · Boolean algebra | accuracy: PCC depth -0.9210 vs length -0.2068 | Hurt |
Closest papers by meaning
- Boule or Baguette? A Study on Task Topology, Length Generalization, and the Benefit of Reasoning Trace (NeurIPS 2026 · similarity 0.89)
- A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers (NeurIPS 2025 · similarity 0.89)
- Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment (ICML 2025 · similarity 0.88)
- Reasoning with Latent Thoughts: On the Power of Looped Transformers (ICLR 2025 · similarity 0.88)
- Recursive Language Models (NeurIPS 2026 · similarity 0.88)
- InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
