AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Linear Transformers Are Secretly Fast Weight Programmers

Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber

ICML 2021 · Architectures and efficiency › Attention and transformers · approach: Attention head and block redesign · 23 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace additive sum writes with a delta update that corrects the current key-value mappingLinear Transformer · WikiText-103, small and medium configurationsperplexity: 37.1 -> 34.1; 38.3 -> 35.5; 31.1 -> 29.7; 33.0 -> 31.5Improved
Replace additive sum writes with a delta updatePerformer · WikiText-103, small and medium configurationsperplexity: 39.0 -> 36.1; 39.6 -> 37.2; 32.2 -> 30.0; 33.8 -> 31.8Improved
Use DPFP projection instead of Performer random featuresLinear attention · WMT14 En-DeBLEU: At d_dot=256: 26.2 vs 24.9 valid and 26.9 vs 25.3 test; at d_dot=512: 26.2 vs 26.7 valid and 27.1 vs 27.7 testMixed
Increase Performer random-feature count to m=256Performer · WMT14 En-DeBLEU: 24.2 -> 26.7 valid; 24.4 -> 27.7 testImproved
Remove absolute positional encoding (ablation)Delta Network · WikiText-103 mediumperplexity: 30.4 -> 29.2 valid; 32.1 -> 31.2 testImproved
Remove additional attention normalization (ablation)Delta Network · WikiText-103 mediumperplexity: 30.4 -> 29.7 valid; 32.1 -> 31.5 testImproved
Remove both absolute positional encoding and additional attention normalization (ablation)Delta Network · WikiText-103 mediumperplexity: 30.4 -> 28.1 valid; 32.1 -> 31.1 testImproved
Remove sum normalization (ablation)Delta Network · WikiText-103, limited-context experimentstraining stabilityHurt
Use a larger projection dimension (ablation)DPFP and Performer · WikiText-103 smallperplexityNo clear effect
Use delta update instead of sum updateDPFP and Performer · WikiText-103 small, non-overcapacityperplexity: Performer: 38.0 -> 36.0 valid, 38.8 -> 37.0 test; DPFP: 37.7 -> 33.9 valid, 38.8 -> 35.0 testImproved
Use DPFP-1 projection instead of the original key representation (ablation)Schlag (2021) update-rule baseline · Synthetic retrieval with repeated keysretrieval lossMixed
Use delta update rather than additive sum updateFast weight memory · Synthetic retrieval with repeated keysretrieval lossImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy