Research map / Architectures and efficiency / RNNs and normalization
Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace additive update with delta rule | Linear Transformer · WikiText-103 test | perplexity: 38.3 -> 35.2 | Improved |
| add recurrent fast net term (Delta RNN) | Delta Net · WikiText-103 test | perplexity: 35.2 -> 35.0 | Improved |
| make fast net an LSTM (Delta LSTM) | Delta Net · WikiText-103 test | perplexity: 35.2 -> 33.8 | Improved |
| add recurrence in slow net (RDN) | Delta Net · WikiText-103 test | perplexity: 35.2 -> 35.2 | No clear effect |
| fast deep MLP generating all layer weights at once | Delta Net · WikiText-103 test | perplexity: 35.2 -> 36.8 | Hurt |
| add recurrent fast net term (Delta RNN) | Delta Net · Code Exec 5 variables | accuracy: 61.4 -> 85.1 | Improved |
| add recurrence in slow net (RDN) | Delta Net · Code Exec 5 variables | accuracy: 61.4 -> 76.3 | Improved |
| add recurrent connections (Delta RNN) | Delta Net · Seq ListOps depth 10 | accuracy: 85.7 -> 83.6 | No clear effect |
| use softmax after fast net sum (Version A) instead of on recurrent input (Version B) (ablation) | Delta RNN · WikiText-103 test | perplexity: 35.0 -> 36.7 | Hurt |
| delay softmax, drop sigmoid on main transform (Version D vs A) (ablation) | Delta LSTM · WikiText-103 test | perplexity: 39.9 -> 33.8 | Improved |
| increase LSTM hidden size 256 to 4,096 (ablation) | LSTM · Seq ListOps depth 15 | accuracy: 24.4 -> 72.2 | Improved |
| double RDN size (4 layers, hidden 256) | RDN · Atari Battlezone 200M | score: 10,980 -> 28,273 | Improved |
Closest papers by meaning
- Real-Time Recurrent Learning using Trace Units in Reinforcement Learning (NeurIPS 2024 · similarity 0.84)
- Rethinking Transformers in Solving POMDPs (ICML 2024 · similarity 0.83)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.83)
- Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps (NeurIPS 2024 · similarity 0.83)
- Transformers Converge to Invariant Algorithmic Cores (NeurIPS 2026 · similarity 0.83)
- Parallelizing Linear Transformers with the Delta Rule over Sequence Length (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
