Research map / Architectures and efficiency / Attention and transformers
A Regularized Framework for Sparse and Structured Neural Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace softmax attention with sparsemax (squared 2-norm regularizer) | textual entailment attention · SNLI test | accuracy: +0.73 over softmax (81.66 to 82.39) | Helped |
| add fused lasso (1-d TV) penalty to sparsemax-style attention (fusedmax) | textual entailment attention · SNLI test | accuracy: +0.02 over sparsemax, +0.75 over softmax | Helped |
| add OSCAR penalty to sparsemax-style attention (oscarmax) | textual entailment attention · SNLI test | accuracy: -0.63 vs sparsemax; +0.10 vs softmax | Mixed |
| add fused lasso penalty to sparsemax-style attention (fusedmax) | sentence summarization attention · DUC 2004 | ROUGE-1: +1.26 over softmax, +0.73 over sparsemax | Helped |
| add OSCAR penalty (oscarmax) | sentence summarization attention · DUC 2004 | ROUGE-1: +0.68 over softmax, +0.15 over sparsemax | Helped |
| add fused lasso penalty (fusedmax) | sentence summarization attention · Gigaword | ROUGE-1: +0.96 over softmax, +0.05 over sparsemax | Helped |
| add fused lasso penalty (fusedmax) | sentence summarization attention · Gigaword | ROUGE-2: -0.16 vs sparsemax; +0.47 vs softmax | Mixed |
| use squared p-norm (p=1.5) regularizer (sq-pnorm-max) | sentence summarization attention · Gigaword | ROUGE-1: +0.81 over softmax | Helped |
| add fused lasso penalty (fusedmax) on NMT | machine translation attention · 10 language pairs NMT | BLEU: all mechanisms within 1 BLEU of best | No clear effect |
Closest papers by meaning
- Sparse Attention as Compact Kernel Regression (NeurIPS 2026 · similarity 0.89)
- SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024 · similarity 0.89)
- Customizing the Inductive Biases of Softmax Attention using Structured Matrices (ICML 2025 · similarity 0.89)
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.89)
- Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models (NeurIPS 2026 · similarity 0.88)
- Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.