Research map / Architectures and efficiency / Attention and transformers
Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective
Etienne Boursier, Claire Boyer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase prompt length L (ablation) | single-layer softmax attention vs normalized linear attention · in-context linear regression, Gaussian inputs, d=5 | output and gradient distance between softmax and linear models | Improved |
| increase prompt length L (ablation) | softmax attention training dynamics · in-context linear regression, isotropic covariates | parameter distance between softmax and linear attention trajectories | Improved |
| use linear attention instead of softmax attention | single-layer attention · in-context linear regression ICL risk at convergence | ICL risk | Improved |
| use anisotropic (Kac-Murdock-Szego) covariates instead of isotropic (ablation) | softmax and linear attention · in-context linear regression ICL risk | convergence of ICL risk to Bayes optimum | Hurt |
| use softmax instead of linear attention | single-layer attention training · in-context linear regression optimization | training stability | Improved |
Closest papers by meaning
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.91)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.90)
- In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness (NeurIPS 2024 · similarity 0.90)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.90)
- In-context Convergence of Transformers (ICML 2024 · similarity 0.90)
- Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
