Research map / Architectures and efficiency / Attention and transformers
Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CHELA, an SSM-attention hybrid that replaces SSMs with multi-resolution short-long convolutions and uses a tiled, hardware-efficient (FlashAttention-style) linear attention to keep linear complexity while improving accuracy on LRA, speech, pixel-level image, and language modeling.
Problem
Hybrid SSM-attention models are quadratic or, when linearized, lose performance; linear attention is slow in causal settings due to memory I/O and cumsum, and long convolutions are unstable without hand-crafted regularization.
Approach
Combine a tiled divide-and-conquer linear attention kernel (following GLA/FlashAttention) with short-long convolutions (parallel short kernels before a long convolution, reparameterized into one kernel at inference) that stabilize the long convolution and supply structured global context to gated linear attention.
Key results
- Achieves 88.19 average accuracy on LRA across six tasks, versus 85.66 for Mega-chunk and 87.40 for SPADE-chunk, with 5.8x speed over the Transformer on the 4K Text task.
- Reaches 97.98% accuracy on raw Speech Commands SC10 with 493K parameters, above S4 at 97.50%.
- Obtains 16.97 PPL on WikiText-103 (258M params) versus 18.07 for Mega-chunk, and 0.96 bpc on enwik8 (48M params) versus 0.99 for Transformer-VQ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CHELA instead of SPADE-chunk | LRA models · LRA average | accuracy: 87.40 -> 88.19 | Helped |
| use CHELA instead of Mega-chunk | language model · WikiText-103 | PPL: 18.07 -> 16.97 | Helped |
| use CHELA instead of Mega-chunk | language model · WikiText-103 | speed: 48k t/s -> 53k t/s | Helped |
| use CHELA instead of Transformer-VQ | language model · enwik8 | PPL: 0.99 -> 0.96 | Helped |
| use CHELA instead of S4 | raw speech classifier · SpeechCommand-Raw | accuracy: 97.50 -> 97.98 | Helped |
| replace Short-Long Convs with Damped EMA (ablation) | CHELA structured mixer · LRA Text | accuracy: 91.10 -> 90.19 | Hurt |
| replace Short-Long Convs with Damped EMA (ablation) | CHELA structured mixer · LRA Image | accuracy: 91.12 -> 85.80 | Hurt |
| replace Short-Long Convs with Damped EMA (ablation) | CHELA structured mixer · LRA PathX | accuracy: 98.65 -> 93.81 | Hurt |
| replace Short-Long Convs with S4D (ablation) | CHELA structured mixer · LRA Text | accuracy: 91.10 -> 90.85 | Hurt |
| replace Short-Long Convs with S4D (ablation) | CHELA structured mixer · LRA Image | accuracy: 91.12 -> 88.95 | Hurt |
| replace Short-Long Convs with S4D (ablation) | CHELA structured mixer · LRA PathX | accuracy: 98.65 -> 94.29 | Hurt |
| add short convolutions before the long convolution (ablation) | CHELA structured mixer · LRA Text | accuracy: 90.35 -> 91.10 | Helped |
Surprising findings
- Linear attention in a causal hybrid was reported to be slower than S4 in wall-clock time despite its theoretical linear complexity, motivating the hardware-efficient implementation.
Limitations
- Optimal combinations of short convolution kernels are not explored.
- Dynamic, input-dependent short convolutions are left for future work.
- Speedup and results are reported for specific settings (e.g., 4K Text task); broader scaling is not shown.
Experimental setup
Datasets: Long Range Arena (ListOps, Text, Retrieval, Image, Pathfinder, Path-X) Speech Commands SC10 (raw) Sequential CIFAR-10 WikiText-103 enwik8
Benchmarks: Long Range Arena Speech Commands SC10 raw sCIFAR WikiText-103 enwik8
Metrics: accuracy perplexity bits-per-byte training speed peak memory
Baselines: Transformer S4 S4D DSS Mamba Mega-chunk SPADE-chunk Transformer-XL Transformer-VQ Luna-256 Performer Linformer
Builds on: FlashAttention gated linear attention MEGA S4 structural reparameterization (RepVGG) GAU
Closest papers by meaning
- Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention (ICML 2024 · similarity 0.88)
- Gated Linear Attention Transformers with Hardware-Efficient Training (ICML 2024 · similarity 0.87)
- Sequence Accumulation and Beyond: Infinite Context Length on Single GPU and Large Clusters (AAAI 2025 · similarity 0.87)
- Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data (NeurIPS 2025 · similarity 0.87)
- Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels (NeurIPS 2025 · similarity 0.87)
- Loki: Low-rank Keys for Efficient Sparse Attention (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.