Research map / Architectures and efficiency / Attention and transformers
Long-Context Generalization with Sparse Attention
Pavlo Vasylenko, Hugo Pitorro, Andre Martins, Marcos Vinicius Treviso
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add adaptive scaling (ASEntmax over fixed-alpha Entmax) (ablation) | Entmax · MQMTAR 1024x length | exact match accuracy (%): 9.3 -> 95.3 | Improved |
| add adaptive scaling (ASEntmax over Entmax) (ablation) | Entmax · Reverse 8x length | exact match accuracy (%): 0.2 -> 56.7 | Improved |
| add adaptive scaling (ASEntmax over Entmax) (ablation) | Entmax · RULER S-NIAH-1 16K | accuracy: 1.2 -> 97.4 | Improved |
| add adaptive scaling (ASEntmax over Entmax) (ablation) | Entmax · ArXiv perplexity 16K | perplexity: 10.07 -> 10.01 | No clear effect |
| add adaptive scaling (ASEntmax over Entmax) (ablation) | Entmax · Lambada perplexity | perplexity: 47.9 -> 41.6 | Improved |
| replace softmax with alpha-entmax | Softmax · ArXiv perplexity 16K | perplexity: 12.71 -> 10.07 | Improved |
| replace softmax with alpha-entmax | Softmax · Reverse 4x length | exact match accuracy (%): 0.0 -> 28.5 | Improved |
| replace softmax with alpha-entmax | Softmax · Copy 64x length | exact match accuracy (%): 85.5 -> 0.0 | Hurt |
| replace softmax with fixed Top-K (K=32) | Softmax · MQMTAR 4x length | exact match accuracy (%): 100.0 -> 6.2 | Hurt |
| replace SSMax with ASEntmax | SSMax · Copy 64x length | exact match accuracy (%): 95.8 -> 86.6 | Hurt |
| replace SSMax with ASEntmax | SSMax · MQMTAR 1024x length | exact match accuracy (%): 26.7 -> 95.3 | Improved |
| replace softmax with ASEntmax | Softmax · PIQA | accuracy: 65.1 -> 63.8 | No clear effect |
Closest papers by meaning
- AdaSplash: Adaptive Sparse Flash Attention (ICML 2025 · similarity 0.91)
- Curse of High Dimensionality Issue in Transformer for Long Context Modeling (ICML 2025 · similarity 0.90)
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps (NeurIPS 2026 · similarity 0.89)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.89)
- Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling (ICML 2025 · similarity 0.89)
- Selective Attention: Enhancing Transformer through Principled Context Control (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
