Research map / Architectures and efficiency / Attention and transformers
MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Unifies linear transformers, SSMs and linear RNNs as one linear attention form, derives conditions for an optimal linear approximation to softmax attention, and proposes MetaLA (query + dynamic decay, no key, self-augmentation, short conv) that beats prior linear models on MQAR, language modeling, LRA and ImageNet.
Problem
Linear complexity models (linear transformers, SSMs, linear RNNs) replace softmax attention, but the optimal design of such models is unclear; the paper asks what linear attention must satisfy to functionally approximate softmax attention.
Approach
Unify LinFormer/SSM/LinRNN into a general linear attention form with Query, Key, Value and decay; derive necessary conditions (dynamic memory, static approximation, least parameters) showing Query plus dynamic decay suffice and Key is unnecessary; build MetaLA with 1-alpha_t in place of keys, self-augmentation and short convolution.
Key results
- MetaLA$_b$ (1.4B, 300B tokens) reaches 49.22 average on SuperGLUE vs 43.21 for Pythia (0.41B, 15B tokens) and 42.83 for Mamba (1.4B, 100B tokens).
- On LRA, MetaLA averages 86.67, above Transformer (47.81) and Mamba (66.59) and close to the best SSM Mega (88.21).
- On ImageNet-1k MetaLA reaches 75.33 top-1 at 6.1M params vs 72.47 for GLA and 73.39 for Mamba at the same size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MetaLA instead of Mamba | 0.36B language model · SuperGLUE | AVG: 43.96 -> 44.05 | Helped |
| Use MetaLA with tied embedding instead of HGRN | 1.3B language model · SuperGLUE | AVG: 45.60 -> 45.81 | Helped |
| Train MetaLA with 300B tokens instead of using Pythia | 1.4B language model · SuperGLUE | AVG: 44.14 -> 49.22 | Helped |
| Use MetaLA instead of Pythia | 0.36B language model · Commonsense Reasoning | AVG: 42.66 -> 42.52 | Hurt |
| Use MetaLA instead of Mamba | 1.4B language model · Commonsense Reasoning | AVG: 47.37 -> 47.01 | Hurt |
| Train MetaLA with 300B tokens instead of using Pythia | 1.4B language model · Commonsense Reasoning | AVG: 49.85 -> 49.99 | Helped |
| Use MetaLA instead of HGRN | 6M image classifier · ImageNet-1k | Acc: 74.40 -> 75.33 | Helped |
| Use MetaLA instead of HGRN | 23M image classifier · ImageNet-1k | Acc: 80.09 -> 80.14 | Helped |
| Use MetaLA instead of Mega | MetaLA · LRA | AVG.: 88.21 -> 86.67 | Hurt |
| Add self-augmentation (ablation) | MetaLA · Commonsense Reasoning, 360M model / 15B tokens | AVG: 42.16 -> 42.52 | Helped |
| Add short convolution (ablation) | MetaLA · Commonsense Reasoning, 360M model / 15B tokens | AVG: 38.51 -> 42.52 | Helped |
| Restore the Key matrix (ablation) | MetaLA · Commonsense Reasoning, 360M model / 15B tokens | AVG: 42.52 -> 40.80 | Hurt |
Surprising findings
- Restoring the Key matrix in MetaLA does not improve performance and lowers average zero-shot accuracy (40.80 vs 42.52), supporting that dynamic decay substitutes for keys.
- MetaLA outperforms softmax attention in some zero-shot and LRA results, which the authors flag as questioning a capacity upper bound implied by approximation theory.
Limitations
- Theoretical analysis is limited to a simplified setting (d_k=1 then vector extension) and assumes expressive functions and good inputs.
- Several results are reported with baselines trained under identical conditions at small scale (360M/1.3-1.4B), and some comparisons use different token budgets.
- Some models underperform the softmax Transformer on MQAR (attention achieves >99), so approximation does not close the recall gap.
Experimental setup
Datasets: The Pile MQAR synthetic ImageNet-1k Long Range Arena
Benchmarks: MQAR SuperGLUE Commonsense Reasoning Long Range Arena ImageNet-1k
Metrics: accuracy
Baselines: Mamba GLA RWKV-4 HGRN RetNet Pythia Based DeiT S4 Mega
Builds on: Mamba GLA RetNet HGRN Griffin
Closest papers by meaning
- MVA: Linear Attention with High-order Query-Keys Integration and Multi-level Vocabulary Decomposition (ICML 2025 · similarity 0.90)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.89)
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.88)
- The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry (ICLR 2024 · similarity 0.87)
- STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs (NeurIPS 2026 · similarity 0.87)
- ZeroS: Zero‑Sum Linear Attention for Efficient Transformers (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.