Research map / Architectures and efficiency / Attention and transformers
SEA: Sparse Linear Attention with Estimated Attention Mask
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEA, a test-time linear-complexity attention that estimates a compressed attention map with Performer plus a CNN decoder, builds a top-k sparse mask from it, and is trained by distilling the attention of a quadratic teacher transformer.
Problem
Quadratic cost of transformer self-attention on long sequences; existing linear/sparse attention methods cannot easily distill from a pretrained teacher's attention matrix, often need retraining from scratch, and lose interpretable full attention maps.
Approach
Estimate a compressed T x K attention matrix with kernel-based linear attention (Performer) and a CNN decoder, select grouped top-k entries, interpolate them into a sparse mask over the full T x T matrix, and run sparse attention; the compressed estimate is distilled directly from teacher attention.
Key results
- On Wikitext2 with OPT-1.3B, SEA achieves lower perplexity than the quadratic baseline (teacher PPL 29.2) when k is increased after training, while using roughly half the memory.
- SEA reports 47.7% lower perplexity than Performer and 54.2% less memory than quadratic attention in language modeling.
- At sequence length 8,192, SEA (k=32) uses 81.05% less peak memory than quadratic attention per the paper's text and about 32.72% of its latency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SEA instead of vanilla attention after positional-embedding interpolation | OPT-125M · WikiText2, context length 4096, PPL. (after interp.) | perplexity: 52.96 -> 43.96 | Helped |
| Use SEA instead of vanilla attention after training | OPT-125M · WikiText2, context length 4096, PPL. (trained) | perplexity: 18.96 -> 23.43 | Hurt |
| Use SEA instead of vanilla attention | OPT-125M · WikiText2, context length 4096 | Memory (MB): 1584 -> 448 (28.28%) | Helped |
| Use SEA instead of vanilla attention | OPT-125M · WikiText2, context length 4096 | Latency (ms): 17.86 -> 14.31 (80.12%) | Helped |
| Use SEA instead of Performer | OPT-125M · WikiText2, context length 4096, PPL. (after interp.) | perplexity: 68.29 -> 43.96 | Helped |
| Use SEA instead of Performer | OPT-125M · WikiText2, context length 4096 | Memory (MB): 102.75 -> 448 | Hurt |
| Use SEA instead of Performer | OPT-125M · WikiText2, context length 4096 | Latency (ms): 2.38 -> 14.31 | Hurt |
| Increase k from 96 to 128 after training with query skips fixed at 1 (ablation) | SEA on OPT-125M · WikiText2, context length 4096 | PPL: 22.38 -> 21.38 | Helped |
| Increase k from 96 to 128 after training with query skips fixed at 1 (ablation) | SEA on OPT-125M · WikiText2, context length 4096 | Latency (ms): 18.12 -> 20.47 | Hurt |
| Use FlatCSR instead of COO | Sparse attention operation · Random inputs | Latency (ms): 75.66 -> 11.4 | Helped |
| Use FlatCSR instead of COO | Sparse attention operation · Random inputs | Memory (MB): 1194 -> 817.5 | Helped |
Surprising findings
- Increasing k after training makes SEA outperform the vanilla quadratic model on Wikitext2 perplexity.
Limitations
- Latency is higher than some linear baselines (Performer, Cosformer, Sinkhorn) in the reported experiments.
- Uniform interpolation for the case where kK < T is not optimal, as the paper acknowledges.
- Evaluation limited to BERT and OPT on GLUE and Wikitext2.
Experimental setup
Datasets: Wikitext2 GLUE MNLI GLUE CoLA GLUE MRPC OpenWebText
Benchmarks: Wikitext2 GLUE MNLI CoLA MRPC OpenWebText
Metrics: perplexity accuracy latency (ms) peak memory (MB) FLOPs
Baselines: Performer Reformer Sinkhorn Cosformer Synthesizer FlashAttention Vanilla quadratic attention
Builds on: Performer knowledge distillation Reformer FlashAttention
Closest papers by meaning
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.89)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.89)
- Sparse Attention as Compact Kernel Regression (NeurIPS 2026 · similarity 0.89)
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.89)
- Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction (NeurIPS 2025 · similarity 0.88)
- Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.