Research map / Architectures and efficiency / Attention and transformers
Simple Hardware-Efficient Long Convolutions for Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Squash regularization to long convolution kernels (ablation) | Long Convs · LRA | accuracy: 69.5 -> 86.1 | Helped |
| Use Smooth instead of Squash regularization (ablation) | Long Conv, random initialization · LRA Avg | accuracy: 86.1 -> 71.8 | Hurt |
| Add Smooth on top of Squash regularization (ablation) | Long Conv, random initialization · LRA Avg | accuracy: 86.1 -> 73.7 | Hurt |
| Use geometric instead of random initialization with Squash (ablation) | Long Conv · LRA Avg | accuracy: 86.1 -> 86.6 | No clear effect |
| Use geometric instead of random initialization (ablation) | Long Conv · sCIFAR | accuracy: 91.0 -> 92.1 | Helped |
| Replace S4 layers with 2D long convolutions | S4ND-ISO · CIFAR | accuracy: 89.9 -> 89.1 | Hurt |
| Use FlashButterfly instead of block-sparse FlashAttention | Path256 model · Path256 (64K) | accuracy: 63.1 -> 92.2 | Helped |
| Use FlashButterfly instead of cuFFT-based convolution | Long convolution · Sequence lengths 1K to 128K | speedup: 2.2$\times$ | Helped |
| Learn Butterfly matrices with block size 16 instead of using fixed Butterfly matrices (ablation) | Sequential CIFAR model · sCIFAR | accuracy: 91.0 -> 91.8 | Helped |
| Increase learned Butterfly block size from 16 to 32 (ablation) | Sequential CIFAR model · sCIFAR | accuracy: 91.8 -> 92.4 | Helped |
| Increase learned Butterfly block size from 32 to 256 (ablation) | Sequential CIFAR model · sCIFAR | speedup: 0.9$\times$ -> 0.6$\times$ | Hurt |
| Replace Transformer MLPs with the learned Butterfly extension | GPT-2-Small · WikiText103 | PPL: 20.6 -> 20.4 | Helped |
Closest papers by meaning
- FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores (ICLR 2024 · similarity 0.90)
- Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond (ICLR 2025 · similarity 0.88)
- Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences (ICML 2024 · similarity 0.88)
- Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling (NeurIPS 2024 · similarity 0.86)
- Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection (NeurIPS 2025 · similarity 0.86)
- Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.