Research map / Architectures and efficiency / RNNs and normalization
Online learning of long-range dependencies
Nicolas Zucchet, Robert Meier, Simon Schug, Asier Mujika, Joao Sacramento
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LRU with dense linear RNN layer (SnAp-1 online) (ablation) | online learning rule · Copy task, 4 layers | final training loss: 8.44 \times 10^{-3} -> 5.82 \times 10^{-1} | Hurt |
| replace LRU with dense linear RNN layer (ablation) | online learning rule · LRA sCIFAR | test accuracy: 79.59 -> 63.71 | Hurt |
| online learning rule vs spatial backprop | online learning rule · LRA sCIFAR | test accuracy: 58.20 -> 79.59 | Improved |
| online learning rule vs 1-step truncated BPTT | online learning rule · LRA IMDB | test accuracy: 84.04 -> 86.48 | Improved |
| online learning rule vs spatial backprop | online learning rule · LRA ListOps | test accuracy: 32.02 -> 37.62 | Improved |
| online rule vs spatial backprop | online learning rule · Copy task, LRU 4 layers | final training loss: 4.66 \times 10^{-1} -> 8.44 \times 10^{-3} | Improved |
| increase GRU depth from 1 to 4 with SnAp-1 (ablation) | SnAp-1 on GRU · Copy task | final training loss: 3.16 \times 10^{-1} -> 3.27 \times 10^{-1} | No clear effect |
| increase GRU depth from 1 to 4 with BPTT (ablation) | BPTT on GRU · Copy task | final training loss: 2.61 \times 10^{-1} -> 1.94 | Improved |
| SnAp-1 sensitivity approximation vs cruder truncated BPTT on linear RNN (ablation) | linear RNN · Copy task | final training loss: 5.81 \times 10^{-1} -> 5.82 \times 10^{-1} | No clear effect |
| SnAp-1 extended to multilayer vs spatial backprop on linear RNN (ablation) | linear RNN · LRA sCIFAR | test accuracy: 50.63 -> 63.71 | Improved |
| shift initial eigenvalue |lambda| distribution toward 1 (ablation) | online learning rule · Copy task | gradient alignment | No clear effect |
| smaller initial |lambda|_min (ablation) | online learning rule · Copy task | training speed | Hurt |
Closest papers by meaning
- A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks (JMLR 2020 · similarity 0.89)
- UnICORNN: A recurrent model for learning very long time dependencies (ICML 2021 · similarity 0.88)
- Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks (ICLR 2024 · similarity 0.88)
- Training biologically plausible recurrent neural networks on cognitive tasks with long-term dependencies (NeurIPS 2023 · similarity 0.88)
- Global Credit Assignment via Dynamical Criticality (ICML 2026 · similarity 0.87)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
