Research map / Architectures and efficiency / Attention and transformers
Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutions
Stefano Massaroli, Michael Poli, Daniel Y Fu, Hermann Kumbong, Rom Nishijima Parnichkun, David W. Romero, Aman Timalsina, Quinn McIntyre, Beidi Chen, Atri Rudra, Ce Zhang, Christopher Re and 2 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-head Hyena (8 heads, tied long filters) instead of Hyena | Hyena 153M · The Pile, 5B tokens | perplexity: 13.3 -> 12.1 | Improved |
| use multi-head Hyena instead of Hyena | Hyena 153M · The Pile, 15B tokens | perplexity: 11.1 -> 10.6 | Improved |
| use multi-head Hyena instead of Hyena | Hyena 355M · The Pile, 15B tokens | perplexity: 9.2 -> 8.9 | Improved |
| use multi-head Hyena instead of Hyena, full 300B-token training | Hyena small · The Pile, 300B tokens | perplexity: 9.3 -> 8.7 | Improved |
| distill MultiHyena filters to state order 16 (ablation) | MultiHyena 154M · LAMBADA | accuracy: 43.2 -> 43.1 | No clear effect |
| distill MultiHyena filters to state order 16 (ablation) | MultiHyena 154M · PIQA | accuracy: 64.6 -> 64.7 | No clear effect |
| distill MultiHyena filters to state order 8 instead of 16 (ablation) | MultiHyena 154M · LAMBADA | accuracy: 43.1 -> 0.0 | Hurt |
| distill MultiHyena filters to state order 8 instead of 16 (ablation) | MultiHyena 154M · PIQA | accuracy: 64.7 -> 51.5 | Hurt |
| distill to state order 4 instead of 8 (ablation) | MultiHyena 154M · HellaSwag | normalized accuracy: 32.7 -> 26.4 | Hurt |
| distill to order 8 instead of 16 (ablation) | MultiHyena 154M · Winogrande | accuracy: 52.6 -> 51.8 | Mixed |
| increase SSM state dimension from 32 to 64 (ablation) | LaughingHyena · peak throughput | throughput: 2% reduction | Hurt |
| replace Transformer with distilled recurrent Hyena at 1.3B | LaughingHyena 1.3B · peak generation throughput | throughput: 10x higher | Improved |
Closest papers by meaning
- Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond (ICLR 2025 · similarity 0.90)
- Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling (NeurIPS 2024 · similarity 0.87)
- State-Free Inference of State-Space Models: The *Transfer Function* Approach (ICML 2024 · similarity 0.86)
- LoCoCo: Dropping In Convolutions for Long Context Compression (ICML 2024 · similarity 0.86)
- Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions (NeurIPS 2026 · similarity 0.86)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
