Research map / Probabilistic and scientific modeling / Variational and generative sampling
Sparse and Continuous Attention Mechanisms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add continuous sparsemax attention on top of discrete softmax attention (combined) | biLSTM text classifier · IMDB | accuracy (%): 90.78 -> 91.10 | Helped |
| replace discrete softmax attention with continuous attention (N<<L RBFs) alone | biLSTM text classifier · IMDB | accuracy (%): 90.78 -> 90.20/90.68/90.52 (softmax) | No clear effect |
| use sparsemax instead of softmax for continuous attention (ablation) | continuous attention · IMDB | accuracy (%): N=64: 90.68 -> 89.63; N=128: 90.52 -> 90.90 | Mixed |
| use combined discrete+continuous attention instead of discrete only | biLSTM translation model · IWSLT17 De-En | BLEU: 23.92 -> 24.25 | Helped |
| use 2D continuous softmax attention instead of discrete attention | MCAN VQA · VQA-v2 test-dev | overall accuracy: 65.83 -> 65.96 | No clear effect |
| use 2D continuous sparsemax attention instead of discrete attention | MCAN VQA · VQA-v2 test-dev | overall accuracy: 65.83 -> 65.79 | No clear effect |
Closest papers by meaning
- Sparse Attention as Compact Kernel Regression (NeurIPS 2026 · similarity 0.90)
- MultiMax: Sparse and Multi-Modal Attention Learning (ICML 2024 · similarity 0.87)
- SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024 · similarity 0.86)
- Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems (NeurIPS 2025 · similarity 0.86)
- AdaSplash: Adaptive Sparse Flash Attention (ICML 2025 · similarity 0.86)
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.