Research map / Architectures and efficiency / Attention and transformers
Scaling Attention via Feature Sparsity
Yan Xie, Tiansheng Wen, Tang Da Huang, Bo Chen, Chenyu You, Stefanie Jegelka, Yifei Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace dense QK scoring with SFA k=8 | GPT2-124M (from scratch) · OpenWebText | validation perplexity: 17.29 -> 18.27 | Mixed |
| replace dense QK scoring with SFA k=16 | Qwen3-0.6B (from scratch) · Pile | validation perplexity: 4.66 -> 4.81 | Mixed |
| use SFA instead of halving Q/K hidden size (ablation) | GPT2-124M · OpenWebText | validation perplexity: 20.88 -> 18.27 | Improved |
| use SFA instead of halving Q/K hidden size (ablation) | Qwen3-0.6B · Pile | average zero-shot accuracy: 36.68 -> 38.94 | Improved |
| use SFA k=8 instead of dense attention | GPT2-124M trained on 32k NIAH · Synthetic NIAH 32k | accuracy: 80% -> 82% | Improved |
| raise SFA k from 8 to 16 at 32k (ablation) | GPT2-124M · Synthetic NIAH 32k | decoding speed: 1.3x -> 1.0x | Hurt |
| use SFA k=16 fine-tune instead of dense fine-tune | Qwen3-0.6B · GSM8K | accuracy: 63.42 -> 61.46 | Hurt |
| use SFA k=16 fine-tune instead of dense fine-tune | Qwen3-8B · NIAH 32768 | accuracy: 95 -> 97 | Improved |
| use SFA k=16 fine-tune instead of dense fine-tune | Qwen3-0.6B · Arxiv | accuracy: 41.17 -> 39.14 | Hurt |
| use SFA with k=4 on inference KV cache | SFA · KV-cache memory | KV-cache size: up to ~40% savings | Improved |
Closest papers by meaning
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.89)
- Sparse Attention as Compact Kernel Regression (NeurIPS 2026 · similarity 0.89)
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps (NeurIPS 2026 · similarity 0.89)
- Selective Attention: Enhancing Transformer through Principled Context Control (NeurIPS 2024 · similarity 0.88)
- Block Sparse Flash Attention (NeurIPS 2026 · similarity 0.88)
- AdaSplash: Adaptive Sparse Flash Attention (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
