Research map / Architectures and efficiency / Attention and transformers
FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlashFFTConv, a fused GPU kernel that computes exact FFT convolutions for long sequences via a Monarch matrix decomposition mapped onto tensor cores, plus partial and frequency-sparse convolutions that skip blocks.
Problem
Long convolution models (long filters equal to sequence length) lag Transformers in wall-clock time because FFT convolutions use poorly on modern accelerators: they do not use tensor cores and incur high I/O between memory levels as sequence length grows.
Approach
Rewrites the FFT convolution as an order-p Monarch decomposition of matrix-matrix multiplies broadcast across the sequence (not batch/hidden), so permutations become on-chip transposes and kernels can be fused up to 32K+ sequence length; sparsity in the convolution (partial kernels and frequency-sparse kernels) is realized by skipping blocks of these matmuls.
Key results
- FlashFFTConv speeds up exact FFT convolution up to 7.93x over PyTorch on H100 (gated convolution) and gives 4.4x end-to-end throughput for HyenaDNA-1M.
- Under a fixed compute budget, FlashFFTConv improves Hyena-s-155M perplexity from 13.4 to 11.1 and M2-BERT-base GLUE score from 77.6 to 80.9.
- FlashFFTConv reaches 96.1% accuracy on Path-512 (256K sequence length), where PyTorch runs out of memory; frequency-sparse kernels reach 79% sparsity with no PPL loss on HyenaDNA-1M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FlashFFTConv instead of PyTorch under the same compute budget | M2-BERT-base-110M · GLUE | GLUE score: 77.6 -> 80.9 | Helped |
| Use FlashFFTConv instead of PyTorch under the same compute budget | Hyena-s-155M · Pile | perplexity: 13.4 -> 11.1 | Helped |
| Enable a 256K sequence length with FlashFFTConv | Long convolution model · Path-512 | accuracy | Helped |
| Use FlashFFTConv instead of PyTorch | FFT convolution · Sequence length 1K | forward latency: 1.57 -> 0.24 ms | Helped |
| Remove the tensor-core Monarch decomposition while retaining fusion (ablation) | FlashFFTConv · FFT convolution, sequence length 1K | forward latency: 0.24 -> 0.67 ms | Hurt |
| Fuse gating operations into the convolution | Gated convolution · Sequence length 1K | forward latency: 2.30 -> 0.29 ms | Helped |
| Use FlashFFTConv instead of PyTorch | M2-BERT-base-110M · Sequence length 128 | throughput: 4,480 -> 8,580 seqs/s | Helped |
| Use FlashFFTConv rather than FlashAttention-v2 | Hyena-2.7B compared with GPT-2.7B · Sequence length 16K | throughput: 21.6 -> 32.3 thousands of tokens per second | Helped |
| Shorten the convolution filter to 2K (ablation) | Hyena-s-8K · Hyena-s-8K | perplexity: 13.8 -> 13.8 | No clear effect |
| Extend the pretrained HyenaDNA-1M model to 4M sequence length using partial convolutions (ablation) | HyenaDNA-1M · DNA validation | perplexity: 2.91 -> 2.90 | No clear effect |
| Set 75% of frequency-domain filter entries to zero (ablation) | HyenaDNA-1M · DNA validation | perplexity: 2.91 -> 2.90 | Helped |
| Set 79% of frequency-domain filter entries to zero (ablation) | HyenaDNA-1M · DNA validation | perplexity: 2.91 -> 2.91 | No clear effect |
Surprising findings
- Hyena-2.7B with FlashFFTConv has lower FLOP utilization than FlashAttention-v2 yet higher end-to-end throughput at 2K-16K sequence length, because convolutions need fewer FLOPs.
- Sparsifying up to 75% of pretrained HyenaDNA-1M kernels in frequency space slightly improved perplexity (2.91 to 2.90).
Limitations
- Frequency-sparse and partial convolutions are evaluated only on pretrained or small models; quality under extended training is not established.
- Achieved end-to-end FLOP utilization (56-62%) is lower than FlashAttention-v2 (66-78%) on A100.
- Speedups shrink for long sequences (e.g., 1.37x-1.5x at multi-million length) because of I/O costs between memory levels.
Experimental setup
Datasets: The Pile C4 GLUE Path-X Path-512 HyenaDNA genome data
Benchmarks: GLUE Path-X Path-512 The Pile perplexity HyenaDNA validation perplexity
Metrics: wall-clock time (ms) throughput memory footprint FLOP utilization perplexity GLUE score classification accuracy speedup
Baselines: PyTorch FFT convolution cuFFTdx fusion-only FlashAttention-v2 GPT-2.7B PyTorch convolution end-to-end
Builds on: Monarch FFT decomposition FlashAttention HyenaDNA Hyena M2-BERT
Closest papers by meaning
- Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond (ICLR 2025 · similarity 0.88)
- FlashMask: Efficient and Rich Mask Extension of FlashAttention (ICLR 2025 · similarity 0.88)
- Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels (NeurIPS 2025 · similarity 0.86)
- SFC: Achieve Accurate Fast Convolution under Low-precision Arithmetic (ICML 2024 · similarity 0.86)
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (ICLR 2024 · similarity 0.85)
- Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.