Research map / Architectures and efficiency / Attention and transformers
Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that one-layer, single-head softmax self-attention with rank-1 weight matrices can act as a contextual mapping via the Boltzmann operator, yielding memorization and universal approximation results for two-layer Transformers.
Problem
Prior analyses of Transformer expressivity (universal approximation and memorization) required many layers or heads because softmax was treated as an approximation of hardmax; it is unclear how few self-attention layers suffice to build a contextual mapping.
Approach
Connects softmax attention to the Boltzmann operator (va^T softmax(va)) and shows that rank-1 Q/K/V weights let a single softmax attention layer map each (sequence, token) pair to a distinct context id, while hardmax attention cannot.
Key results
- One-layer rank-1 Transformer reaches nearly perfect memorization of CoNLL-2003 training data within 400 epochs, with 0.9872 accuracy at 800 epochs.
- Hardmax-based one-layer self-attention is proven not to be a contextual mapping.
- Two-layer Transformers with one single-head softmax self-attention layer are universal approximators for continuous permutation equivariant functions on compact domains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 1-layer rank-1 Transformer instead of a deeper rank-1 Transformer (ablation) | Rank-1 Transformer · CoNLL-2003 train | training accuracy: 0.9872 at 800 epochs | Mixed |
| Increase depth from 1 layer to 3 or 6 layers (ablation) | Rank-1 Transformer · CoNLL-2003 train | training accuracy | Helped |
Surprising findings
- Hardmax-based one-layer self-attention cannot distinguish certain input sequences regardless of the number of heads.
- Deeper networks mainly seem to speed up optimization rather than increase theoretical capacity, per the authors' conjecture.
Limitations
- Universal approximation requires a feed-forward quantization layer before self-attention.
- Assumes tokenwise separatedness and no duplicate tokens within a sequence.
- Experiments are small-scale and only show faster training with depth, not a full optimization analysis.
Experimental setup
Datasets: CoNLL-2003
Benchmarks: CoNLL-2003
Metrics: training accuracy
Baselines: Kim et al. 2023 (2n-layer memorization) Yun et al. 2020 (universal approximation Transformers) hardmax self-attention
Builds on: Kim et al. contextual mapping Yun et al. Transformers are universal in-context learners Kolmogorov representation theorem analyses of Transformers
Closest papers by meaning
- Memorization Capacity of Multi-Head Attention in Transformers (ICLR 2024 · similarity 0.89)
- Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling (NeurIPS 2024 · similarity 0.89)
- kNN Attention Demystified: A Theoretical Exploration for Scalable Transformers (ICLR 2025 · similarity 0.88)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.88)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
- InfoFlow: A Framework for Multi-Layer Transformer Analysis (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.