Research map / Architectures and efficiency / Attention and transformers
Hyena Hierarchy: Towards Larger Convolutional Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace attention with Hyena operator (order 2) in ViT | ViT on ImageNet-1k with 16x16 patches · ImageNet-1k classification | top-1 accuracy: 0.0 | No clear effect |
| replace attention with Hyena operator in ViT with 8x8 patches | ViT on ImageNet-1k · ImageNet-1k 8x8 patches | top-1 accuracy: -0.2 | Hurt |
| use Hyena implicit filter parametrization (FFN with decay window, sine) instead of CKConv/TransferFunc/Conv1d for long convolutions (ablation) | order-2 Hyena on associative recall · associative recall, length 131k, vocab 30 | accuracy: +82.9 over CKConv (97.2 vs 14.3) | Helped |
| Hyena operator instead of attention/other subquadratic operators (GSS, H3, AFT, RWKV) | 2-layer models on associative recall · associative recall, length 131k, vocab 30 | accuracy: 97.2 vs at most 2.3 for others | Helped |
| deeper and thinner Hyena (slim, 1.5x depth, FFN width multiplier 2) (ablation) | Hyena-3 on WikiText103 · WikiText103 perplexity | perplexity: -0.1 | Helped |
| Hyena-2 vs GPT at matched parameters on The Pile | 153M and 355M models · The Pile 15B tokens | perplexity: 153M: 11.1 vs GPT 125M 11.2; 355M: 9.2 vs GPT 9.1 | Mixed |
| Hyena vs GPT on SuperGLUE few-shot | 153M Hyena vs GPTNeo 125M · SuperGLUE 3-shot average | average accuracy: 49.3 vs 49.1 (zero-shot: 41.5 vs 43.8) | Mixed |
Closest papers by meaning
- Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions (NeurIPS 2026 · similarity 0.89)
- Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond (ICLR 2025 · similarity 0.88)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.86)
- LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (ICLR 2024 · similarity 0.86)
- Zebra-Llama: Towards Extremely Efficient Hybrid Models (NeurIPS 2025 · similarity 0.86)
- Zoology: Measuring and Improving Recall in Efficient Language Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.