Research map / Architectures and efficiency / Attention and transformers
Hungry Hungry Hippos: Towards Language Modeling with State Space Models
Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, Christopher Re
This paper tried a common change: Add attention module
Add two attention layers to H3 (hybrid H3-attention) · OpenWebText · perplexity (lower better): 20.6 to 19.6 vs Transformer Improved
What 492 accepted papers report for this change
- 444 improved
- 20 no clear effect
- 4 mixed
- 24 hurt
- Multi-Scale Wavelet Transformers for Operator Learning of Dynamical Systems ICML 2026 · add wavelet attention block · CKF step 64 · Rel L2: 1.09 -> 0.35Improved
- DA-DFGAS:Differentiable Federated Graph Neural Architecture Search with Distribution-Aware Attentive Aggregation AAAI 2026 · add attention mechanism to aggregation · Table 4 datasets · parameters: about 5-10% more parametersHurt
- Caracal: Causal Architecture via Spectral Mixing ICML 2026 · add sliding window attention layers to MHF stack · Avg accuracy · avg accuracy (%): 48.22 -> 49.01Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add two attention layers to H3 (hybrid H3-attention) | H3 125M · OpenWebText | perplexity (lower better): 20.6 to 19.6 vs Transformer | Improved |
| Use H3 layer instead of S4D/GSS SSM layers | 125M language model · OpenWebText | perplexity (lower better): 21.0 vs 24.9 (S4D) and 24.0 (GSS) | Improved |
| Pure H3 vs Transformer at 125M | H3 125M · OpenWebText | perplexity (lower better): 21.0 vs 20.6 (within 0.4) | No clear effect |
| Add H3 layer for synthetic associative recall | 2-layer models · Associative Recall synthetic | accuracy: 99.8 vs 86.0 (S4D) and 78.0 (GSS) | Improved |
| Add H3 layer for synthetic induction head | 2-layer models · Induction Head synthetic | accuracy: 100.0 vs 35.6 (S4D) and 6.8 (GSS) | Improved |
| FlashConv (fused block FFT + state passing) applied to S4 | S4 · Long Range Arena | speedup over Transformer: 2.9× to 5.8× | Improved |
| Hybrid H3-attention instead of Transformer at 1.3B inference | 1.3B language model · Inference throughput, prompt length 1536 | tokens/s (higher better): 1240 vs 520 (about 2.4×) | Improved |
| Hybrid H3-attention vs Transformer at 2.7B on Pile scale | 2.7B language model · Pile perplexity | perplexity (lower better): 5.4 vs 5.7 (GPT-Neo-2.7B) | Improved |
| Hybrid H3-attention vs Transformer on SuperGLUE zero-shot average at 2.7B | 2.7B language model · SuperGLUE zero-shot | average accuracy (higher better): 56.8 vs 55.5 (OPT-2.7B) and 54.6 (GPT-Neo-2.7B) | Improved |
| Hybrid H3 with 2 attention layers vs pure H3 at 125M (ablation) | 125M language model · OpenWebText | perplexity (lower better): 19.6 vs 21.0 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- FlashAttention — FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (NeurIPS 2022) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Implicit Language Models are RNNs: Balancing Parallelization and Expressivity (ICML 2025 · similarity 0.90)
- Quamba: A Post-Training Quantization Recipe for Selective State Space Models (ICLR 2025 · similarity 0.90)
- The Expressive Capacity of State Space Models: A Formal Language Perspective (NeurIPS 2024 · similarity 0.90)
- Longhorn: State Space Models are Amortized Online Learners (ICLR 2025 · similarity 0.90)
- Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning (NeurIPS 2025 · similarity 0.89)
- Repeat After Me: Transformers are Better than State Space Models at Copying (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
