Research map / Architectures and efficiency / Attention and transformers
Linear Log-Normal Attention with Unbiased Concentration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Linear Log-Normal (LLN) Attention, a linear-complexity attention whose exponential feature maps are moment-matched to softmax attention's log-normal distribution, entropy, and spectral gap, and evaluates it on GLUE with RoBERTa-base.
Problem
Quadratic time and memory of softmax self-attention limits long-sequence Transformers, and existing linearized attention methods often underperform softmax attention; the paper seeks to understand what makes softmax attention work and design a linear attention that matches it.
Approach
Model softmax attention entries as log-normal with a temperature set by query/key variance, measure concentration with entropy (biased) and spectral gap (unbiased), and design a linear attention using exponential feature maps with scaling parameters alpha and beta chosen by moment matching so its distribution and concentration curves mimic softmax attention; optionally average with block-diagonal
Key results
- LLN+Diag Attention reaches 86.9% average accuracy across MNLI, QNLI, QQP, and SST-2, matching the 87.0% softmax baseline and beating other linear attention methods such as T1 (86.46%) and Flash (86.52%) on average, though Nyströmformer reports 86.8%.
- LLN Attention without the diagonal component reaches 85.4% average, above ELU (82.8%) and Performer (70.6%).
- Memory and speed tables show LLN Attention scaling linearly and handling sequences at least four times longer than softmax attention before softmax runs out of memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add block-diagonal attention to LLN Attention (ablation) | LLN Attention · MNLI | accuracy: 77.0 -> 80.0 | Helped |
| Add block-diagonal attention to LLN Attention (ablation) | LLN Attention · QNLI | accuracy: 85.1 -> 86.5 | Helped |
| Add block-diagonal attention to LLN Attention (ablation) | LLN Attention · QQP | accuracy: 88.9 -> 89.7 | Helped |
| Add block-diagonal attention to LLN Attention (ablation) | LLN Attention · SST-2 | accuracy: 90.6 -> 91.6 | Helped |
| Add block-diagonal attention to LLN Attention (ablation) | LLN Attention · GLUE average | accuracy: 85.4 -> 86.9 | Helped |
| Use moment matching for LLN Attention instead of α=β=1 (ablation) | LLN Attention · attention matrix variance | variance | Helped |
| Use LLN Attention instead of Softmax Attention | RoBERTa-base · GLUE average | accuracy: 87.0 -> 85.4 | Hurt |
| Use LLN+Diag Attention instead of Softmax Attention | RoBERTa-base · GLUE average | accuracy: 87.0 -> 86.9 | No clear effect |
| Use LLN Attention instead of Nyströmformer | RoBERTa-base · RoBERTa-base, sequence length 16384 | time [sec/it]: 16.7 -> 11.8 | Helped |
Surprising findings
- Without moment matching, the variance of LLN attention is much smaller than softmax attention's, showing that plain exponential feature maps do not reproduce softmax concentration.
Limitations
- Short-range interactions can be diluted by linear attention, so a block-diagonal softmax component is needed to recover performance.
- Analysis relies on Gaussian and zero-mean assumptions for queries and keys, and the log-normal approximation is valid only for moderate or broad variance regimes.
- Evaluation is limited to RoBERTa-base on GLUE subsets; the paper does not report large-scale language model results.
Experimental setup
Datasets: WikiText-103 GLUE MNLI QNLI QQP SST-2 Long Range Arena
Benchmarks: GLUE MNLI GLUE QNLI GLUE QQP GLUE SST-2
Metrics: accuracy training time per iteration memory usage
Baselines: Softmax Attention Nyströmformer Performer ELU linear attention Longformer Transformer LS CosFormer Flash
Builds on: Linear Transformers Nystromformer Performer RoBERTa
Closest papers by meaning
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.88)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.87)
- Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems (NeurIPS 2025 · similarity 0.87)
- Graph Convolutions Enrich the Self-Attention in Transformers! (NeurIPS 2024 · similarity 0.87)
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.87)
- ZeroS: Zero‑Sum Linear Attention for Efficient Transformers (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.