AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Going Beyond Linear Transformers with Recurrent Fast Weight Programmers

Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber

NeurIPS 2021 · Architectures and efficiency › RNNs and normalization · approach: Brain-inspired recurrent architectures · 13 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace additive update with delta ruleLinear Transformer · WikiText-103 testperplexity: 38.3 -> 35.2Improved
add recurrent fast net term (Delta RNN)Delta Net · WikiText-103 testperplexity: 35.2 -> 35.0Improved
make fast net an LSTM (Delta LSTM)Delta Net · WikiText-103 testperplexity: 35.2 -> 33.8Improved
add recurrence in slow net (RDN)Delta Net · WikiText-103 testperplexity: 35.2 -> 35.2No clear effect
fast deep MLP generating all layer weights at onceDelta Net · WikiText-103 testperplexity: 35.2 -> 36.8Hurt
add recurrent fast net term (Delta RNN)Delta Net · Code Exec 5 variablesaccuracy: 61.4 -> 85.1Improved
add recurrence in slow net (RDN)Delta Net · Code Exec 5 variablesaccuracy: 61.4 -> 76.3Improved
add recurrent connections (Delta RNN)Delta Net · Seq ListOps depth 10accuracy: 85.7 -> 83.6No clear effect
use softmax after fast net sum (Version A) instead of on recurrent input (Version B) (ablation)Delta RNN · WikiText-103 testperplexity: 35.0 -> 36.7Hurt
delay softmax, drop sigmoid on main transform (Version D vs A) (ablation)Delta LSTM · WikiText-103 testperplexity: 39.9 -> 33.8Improved
increase LSTM hidden size 256 to 4,096 (ablation)LSTM · Seq ListOps depth 15accuracy: 24.4 -> 72.2Improved
double RDN size (4 layers, hidden 256)RDN · Atari Battlezone 200Mscore: 10,980 -> 28,273Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy