Research map / Architectures and efficiency / Attention and transformers
Scatterbrain: Unifying Sparse and Low-rank Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add sparse (LSH) component to low-rank Performer attention (Scatterbrain) | Performer · T2T-ViT drop-in ImageNet | top-1 accuracy (%): 80.1% -> 80.7% | Helped |
| add Performer low-rank component to sparse SMYRF | SMYRF · T2T-ViT drop-in ImageNet | avg approx error: 11.4% -> 5.3% | Helped |
| discount low-rank contribution on sparse support vs naive SMYRF + Performer sum (ablation) | naive sparse+low-rank · T2T-ViT drop-in ImageNet | avg approx error: 12.6% -> 5.3% | Helped |
| Scatterbrain vs Reformer on language modeling | Reformer · WikiText-103 | perplexity: 27.68 -> 26.72 | Helped |
| Scatterbrain vs Performer on language modeling | Performer · WikiText-103 | perplexity: 66 -> 26.72 | Helped |
| Scatterbrain vs Reformer on Copy | Reformer · Copy | perplexity: 6.8 -> 2.58 | Helped |
| Scatterbrain vs Performer on LRA average | Performer · LRA | accuracy (%): 57.05 -> 59.38 | Helped |
| Scatterbrain vs Reformer on LRA average | Reformer · LRA | accuracy (%): 53.9 -> 59.38 | Helped |
| Scatterbrain vs full attention on LRA average at 1/8 parameters | Full Attention · LRA | accuracy (%): 59.62 -> 59.38 | No clear effect |
| Scatterbrain vs Performer on LRA Pathfinder | Performer · LRA Pathfinder | accuracy (%): 68.6 -> 69.91 | Helped |
| Scatterbrain vs full attention | Full Attention · T2T-ViT drop-in ImageNet | top-1 accuracy (%): 81.7% -> 80.7% | Hurt |
Closest papers by meaning
- Loki: Low-rank Keys for Efficient Sparse Attention (NeurIPS 2024 · similarity 0.85)
- Expanding Sparse Tuning for Low Memory Usage (NeurIPS 2024 · similarity 0.85)
- CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models (ICML 2025 · similarity 0.85)
- SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference (ICML 2025 · similarity 0.85)
- From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit (NeurIPS 2025 · similarity 0.84)
- SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.