Research map / Architectures and efficiency / Attention and transformers
Linear Transformers Are Secretly Fast Weight Programmers
Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace additive sum writes with a delta update that corrects the current key-value mapping | Linear Transformer · WikiText-103, small and medium configurations | perplexity: 37.1 -> 34.1; 38.3 -> 35.5; 31.1 -> 29.7; 33.0 -> 31.5 | Improved |
| Replace additive sum writes with a delta update | Performer · WikiText-103, small and medium configurations | perplexity: 39.0 -> 36.1; 39.6 -> 37.2; 32.2 -> 30.0; 33.8 -> 31.8 | Improved |
| Use DPFP projection instead of Performer random features | Linear attention · WMT14 En-De | BLEU: At d_dot=256: 26.2 vs 24.9 valid and 26.9 vs 25.3 test; at d_dot=512: 26.2 vs 26.7 valid and 27.1 vs 27.7 test | Mixed |
| Increase Performer random-feature count to m=256 | Performer · WMT14 En-De | BLEU: 24.2 -> 26.7 valid; 24.4 -> 27.7 test | Improved |
| Remove absolute positional encoding (ablation) | Delta Network · WikiText-103 medium | perplexity: 30.4 -> 29.2 valid; 32.1 -> 31.2 test | Improved |
| Remove additional attention normalization (ablation) | Delta Network · WikiText-103 medium | perplexity: 30.4 -> 29.7 valid; 32.1 -> 31.5 test | Improved |
| Remove both absolute positional encoding and additional attention normalization (ablation) | Delta Network · WikiText-103 medium | perplexity: 30.4 -> 28.1 valid; 32.1 -> 31.1 test | Improved |
| Remove sum normalization (ablation) | Delta Network · WikiText-103, limited-context experiments | training stability | Hurt |
| Use a larger projection dimension (ablation) | DPFP and Performer · WikiText-103 small | perplexity | No clear effect |
| Use delta update instead of sum update | DPFP and Performer · WikiText-103 small, non-overcapacity | perplexity: Performer: 38.0 -> 36.0 valid, 38.8 -> 37.0 test; DPFP: 37.7 -> 33.9 valid, 38.8 -> 35.0 test | Improved |
| Use DPFP-1 projection instead of the original key representation (ablation) | Schlag (2021) update-rule baseline · Synthetic retrieval with repeated keys | retrieval loss | Mixed |
| Use delta update rather than additive sum update | Fast weight memory · Synthetic retrieval with repeated keys | retrieval loss | Improved |
Closest papers by meaning
- Learning Linear Attention in Polynomial Time (NeurIPS 2025 · similarity 0.86)
- Mixture-of-Top-$k$ Attention: Efficient Attention as Scalable Fast Weights (NeurIPS 2026 · similarity 0.86)
- On the Duality between Gradient Transformations and Adapters (ICML 2025 · similarity 0.86)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.85)
- Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformers (NeurIPS 2025 · similarity 0.85)
- Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
