AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modules

Sarthak Mittal, Alex Lamb, Anirudh Goyal, Vikram Voleti, Murray Shanahan, Guillaume Lajoie, Michael Mozer, Yoshua Bengio

ICML 2020 · Architectures and efficiency › Attention and transformers · approach: Sparse and structured attention · 15 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add bidirectional top-down attention to the hierarchical modular recurrent architecture (ablation)Hierarchical RIMs · Sequential MNIST 19×19test accuracy (%): 72.0 -> 74.2Improved
Add bidirectional top-down attention to the hierarchical modular recurrent architecture (ablation)Hierarchical RIMs · Sequential MNIST 24×24test accuracy (%): 50.3 -> 51.4Improved
Add bidirectional top-down attention to the hierarchical modular recurrent architecture (ablation)Hierarchical RIMs · Sequential CIFAR 32×32test accuracy (%): 46.8 -> 52.2Improved
Add bidirectional top-down attention to the hierarchical modular recurrent architecture (ablation)Hierarchical RIMs · WikiText-103 testperplexity: 38.1 -> 36.8Improved
Add bidirectional top-down attention to the hierarchical modular recurrent architecture (ablation)Hierarchical RIMs · Sequential CIFAR 19×19test accuracy (%): 57.2 -> 60.1Improved
Use bidirectional top-down information flow (ablation)LSTM with hierarchy · Sequential CIFAR 32×32test accuracy (%): 33.2 -> 42.1Improved
Add attention to the hierarchical LSTM (ablation)LSTM with hierarchy · Sequential CIFAR 32×32test accuracy (%): 33.2 -> 40.0Improved
Add bidirectional top-down connections to the hierarchical attention LSTM (ablation)LSTM with hierarchy and attention · Sequential CIFAR 32×32test accuracy (%): 40.0 -> 43.0Improved
Add bidirectional top-down connections to the hierarchical attention LSTM (ablation)LSTM with hierarchy and attention · Sequential MNIST 24×24test accuracy (%): 31.0 -> 20.8Hurt
Use BRIMs instead of LSTMRecurrent policy network · Atari Asteroidsscore: 3801 -> 2000Hurt
Use BRIMs instead of LSTMRecurrent policy network · Atari Alienscore: 1612 -> 4102Improved
Use BRIMs instead of LSTMRecurrent model · Adding, 1000 train/2000 test sequence length, 10 valueserror: 17.246 -> 11.90Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy