Research map / Architectures and efficiency / Attention and transformers
To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Models
Eran Malach, Omid Saremi, Sinead Williamson, Arwen Bradley, Aryo Lotfi, Emmanuel Abbe, Joshua M. Susskind, Etai Littwin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on interactive tool-use trajectories (pointer memory) instead of no-tool or single-turn (ablation) | Mamba · multi-digit addition, train 5 digits | accuracy: 5-digit training to 1,000-digit addition | Improved |
| remove CoT, remove tool-use, or use single-turn tool-use (ablation) | Mamba · multi-digit addition | length generalization | Hurt |
| use SSM/RNN (Mamba) instead of Transformer (Pythia) with interactive tool-use | multiplication n x 1 · train up to 10 digits | accuracy at largest length: Mamba 10->1K (100%) vs Pythia 10->20 (79%) | Improved |
| use SSM/RNN instead of Transformer with interactive tool-use | Logical Graph · train up to 10 nodes | accuracy at 1K nodes: Mamba 98% vs Pythia 5% | Improved |
| use Mamba with tools on Tower of Hanoi | Mamba · Tower of Hanoi, train 8 disks | accuracy: 8->12 (49%) | Mixed |
| co-train multiplication with auxiliary addition task under 250 training steps (ablation) | Mamba multiplication · train up to 20 digits | length generalization | Improved |
| co-train multiplication with auxiliary addition task at 500 steps (ablation) | Mamba multiplication · train up to 20 digits | length generalization | Improved |
| co-train multiplication with auxiliary addition task at 800 steps (ablation) | Mamba multiplication · train up to 20 digits | length generalization | No clear effect |
| train on interactive agent trajectories instead of single-turn agent (ablation) | Mamba-1.4B · code fixing on larger codebases than training | pass rate | Improved |
| use Transformer (Pythia-1.4B) instead of Mamba-1.4B on small codebases with distillation | code fixing · small codebases, distillation agent | pass rate: over 90% | Improved |
Closest papers by meaning
- Longhorn: State Space Models are Amortized Online Learners (ICLR 2025 · similarity 0.90)
- Repeat After Me: Transformers are Better than State Space Models at Copying (ICML 2024 · similarity 0.89)
- On the Expressiveness and Length Generalization of Selective State Space Models on Regular Languages (AAAI 2025 · similarity 0.88)
- Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theory (ICLR 2025 · similarity 0.88)
- Universal Length Generalization with Turing Programs (ICML 2025 · similarity 0.88)
- From Generalization Analysis to Optimization Designs for State Space Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
