Research map / Architectures and efficiency / Attention and transformers
Untangling tradeoffs between recurrence and self-attention in artificial neural networks
Giancarlo Kerg, Bhargav Kanuparthi, Anirudh Goyal ALIAS PARTH GOYAL, Kyle Goyette, Yoshua Bengio, Guillaume Lajoie
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add relevancy screening (RelRNN vs. sparse attention SAB without screening) | RNN with sparse attention · Transfer Copy T=5000 | accuracy (%): 95 -> 99 | Improved |
| add relevancy-screened attention to LSTM (RelLSTM vs LSTM) | LSTM · Transfer Copy T=5000 | accuracy (%): 14 -> 99 | Improved |
| add relevancy screening to RNN (RelRNN vs expRNN) | RNN · Transfer Copy T=5000 | accuracy (%): 50 -> 99 | Improved |
| use full self-attention (MemRNN) instead of screened sparse attention | RelRNN · Transfer Copy T=5000 | accuracy (%): 99 -> OOM | Hurt |
| add relevancy screening (RelLSTM vs LSTM) | LSTM · Denoise T=2000 | accuracy (%): 15 -> 99 | Improved |
| add relevancy screening to RNN (RelRNN vs orth-RNN) | RNN · Denoise T=2000 | accuracy (%): 3 -> 99 | Improved |
| use full attention MemRNN instead of RelRNN at long T | RelRNN · Denoise T=2000 | accuracy (%): 99 -> OOM | Hurt |
| add relevancy screening to RNN (RelRNN vs RNN) | RNN · PTB char-level | BPC: 1.56 -> 1.43 | Improved |
| add relevancy screening to RNN (RelRNN vs RNN) | RNN · PTB char-level | accuracy (%): 66 -> 69 | Improved |
| add relevancy screening to LSTM (RelLSTM vs LSTM) | LSTM · PTB char-level | BPC: 1.36 -> 1.36 | No clear effect |
| add relevancy screening to LSTM (RelLSTM vs LSTM) | LSTM · pMNIST | accuracy (%): 91.1 -> 94.3 | Improved |
| add relevancy screening to RNN (RelRNN vs expRNN) | RNN · pMNIST | accuracy (%): 96.6 -> 92.8 | Hurt |
Closest papers by meaning
- Self-Attention-Based Contextual Modulation Improves Neural System Identification (ICLR 2025 · similarity 0.88)
- The emergence of sparse attention: impact of data distribution and benefits of repetition (NeurIPS 2025 · similarity 0.88)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.87)
- Understanding the Differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks (NeurIPS 2024 · similarity 0.87)
- Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers (ICML 2025 · similarity 0.87)
- Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
