Research map / Architectures and efficiency / Attention and transformers
Understanding Dynamic Compute Allocation in Recurrent Transformers
Ibraheem Muhammad Moosa, Suhas Lohit, Ye Wang, Moitreya Chatterjee, Wenpeng Yin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace fixed depth 8 with ANIRA-E adaptive depth | Recurrent-Llama-3.2 · GSM-Symbolic Main | exact accuracy (%): 46.26 -> 45.70 | No clear effect |
| replace fixed depth 8 with ANIRA-O adaptive depth | Recurrent-Llama-3.2 · GSM-Symbolic Main | exact accuracy (%): 46.26 -> 43.68 | Hurt |
| replace fixed depth 8 with ANIRA-E adaptive depth | Recurrent-Llama-3.2 · GSM-Symbolic P1 | exact accuracy (%): 23.54 -> 24.06 | No clear effect |
| replace fixed depth 8 with ANIRA-O adaptive depth | Recurrent-Llama-3.2 · GSM-Symbolic P2 | exact accuracy (%): 5.80 -> 5.08 | Hurt |
| use ANIRA-O instead of fixed depth 8: compute reduction | Recurrent-Llama-3.2 · GSM-Symbolic Main | mean allocated depth: 8 -> 4.108 | Improved |
| use ANIRA-E instead of fixed depth 8: compute reduction | Recurrent-Llama-3.2 · GSM-Symbolic Main | mean allocated depth: 8 -> 6.060 | Improved |
| harder GSM-Symbolic split (P2 vs Main) with ANIRA-E | ANIRA-E · GSM-Symbolic | mean allocated depth: 6.060 -> 5.982 | No clear effect |
| harder GSM-Symbolic split (P2 vs Main) with ANIRA-O | ANIRA-O · GSM-Symbolic | mean allocated depth: 4.108 -> 4.001 | No clear effect |
| online halting (ANIRA-O) instead of ANIRA-E allocation rule | ANIRA · BREVO | feature explaining compute: DFS traversal depth coefficient: -0.005 -> +0.302 | Mixed |
| categorical graph-size effects instead of linear N in the regression (ablation) | ANIRA-E · BREVO compute regression | R^2 (%): 68.66 -> 73.37 | Improved |
| categorical graph-size effects instead of linear N in the regression (ablation) | ANIRA-O · BREVO compute regression | R^2 (%): 70.00 -> 70.88 | No clear effect |
| adaptive depth allocation (ANIRA-O) for extrapolation to longer MANO expressions | ANIRA-O · MANO extrapolation | accuracy | Mixed |
Closest papers by meaning
- Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation (NeurIPS 2025 · similarity 0.90)
- Distilling Sequential Computation in Transformer Language Models (NeurIPS 2026 · similarity 0.89)
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (NeurIPS 2025 · similarity 0.88)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.88)
- D-LLM: A Token Adaptive Computing Resource Allocation Strategy for Large Language Models (NeurIPS 2024 · similarity 0.88)
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
