Research map / Architectures and efficiency / Attention and transformers
Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models
John Cooper, Ilias Diakonikolas, Mingchen Ma, Frederic Sala
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| put SSM layer before the Transformer layer in a 2-layer hybrid (ablation) | hybrid on Selective Copy (~2000 params) · Selective Copy | accuracy: 0.433 -> 0.999 | Improved |
| use hybrid instead of pure TF or SSM at ~2000 params | 2-layer models · Selective Copy | accuracy: 0.352 / 0.305 -> 0.999 | Improved |
| use hybrid instead of pure TF or SSM at ~6000 params | 2-layer models · Multi-Key Associative Recall | accuracy: 0.230 / 0.356 -> 0.990 | Improved |
| put Transformer layer before SSM layer in the hybrid (ablation) | hybrid on Multi-Key Associative Recall (~6000 params) · Multi-Key Associative Recall | accuracy: 0.990 -> 0.286 | Hurt |
| use hybrid instead of pure models on associative recall with decoding (3 layers, ~1M params) | 3-layer models · Associative Recall with Decoding | accuracy: <0.4 -> >0.5 | Improved |
| use hybrid instead of SSM or Transformer for Needle-in-a-Haystack at small scale | 2-layer models, full-context attention · Needle in a Haystack | accuracy | Improved |
| use hybrid instead of pure Transformer for longer test sequences | ~100M-param models trained on lengths 20-50 · Associative Recall with Decoding length generalization | accuracy: within 2% on short sequences -> around 10% gap on long | Improved |
| use hybrid instead of SSM/TF under shifted training bit proportions (train 0.3) | 12-layer models · Associative Recall with Decoding OOD, test proportion 0.2 | accuracy: 0.17 / 0.64 -> 0.74 | Improved |
| use hybrid with high training bit proportion 0.9 | 12-layer models · Associative Recall with Decoding OOD, test proportion 0.2 | accuracy: 0.86 (SSM) vs 0.80 (hybrid) | Hurt |
| increase model size from ~2000 to ~12000 params for pure Transformer (ablation) | pure TF on Selective Copy · Selective Copy | accuracy: 0.352 -> 0.923 | Improved |
Closest papers by meaning
- Repeat After Me: Transformers are Better than State Space Models at Copying (ICML 2024 · similarity 0.89)
- Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling (NeurIPS 2024 · similarity 0.88)
- Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformers (NeurIPS 2025 · similarity 0.88)
- B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory (NeurIPS 2024 · similarity 0.88)
- Selective Induction Heads: How Transformers Select Causal Structures in Context (ICLR 2025 · similarity 0.87)
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
