Research map / Architectures and efficiency / RNNs and normalization
Learning Hierarchical Information Flow with Recurrent Neural Modules
Danijar Hafner, Alexander Irpan, James Davidson, Nicolas Heess
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ThalNet instead of a standard GRU baseline | Language modeling · text8 | BPC: 1.41 -> 1.39 | Improved |
| Use ThalNet instead of a stacked GRU baseline | Sequential image classification · Sequential Permuted MNIST | test accuracy | Improved |
| Use ThalNet instead of a stacked GRU baseline | Sequential image classification · Sequential CIFAR-10 | test accuracy | Improved |
| Use recurrent modules instead of fully connected-only modules (ablation) | ThalNet · Sequential MNIST | training speed and stability | Improved |
| Add a fully connected layer before the GRU module (ablation) | ThalNet · Sequential MNIST | test accuracy | Improved |
| Add a fully connected layer after the GRU module (ablation) | ThalNet · Sequential MNIST | test accuracy | No clear effect |
| Use weight-normalized reading instead of linear or fast-softmax reading (ablation) | ThalNet · Sequential MNIST | training stability | Improved |
| Use fast Gaussian reading instead of the other tested reading mechanisms (ablation) | ThalNet · Sequential MNIST | test accuracy | Hurt |
Closest papers by meaning
- Efficient recurrent architectures through activity sparsity and sparse back-propagation through time (ICLR 2023 · similarity 0.84)
- Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (ICML 2018 · similarity 0.84)
- The Uncanny Similarity of Recurrence and Depth (ICLR 2022 · similarity 0.84)
- Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modules (ICML 2020 · similarity 0.83)
- Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons (NeurIPS 2026 · similarity 0.83)
- Complex priors and flexible inference in recurrent circuits with dendritic nonlinearities (ICLR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
