Research map / Architectures and efficiency / RNNs and normalization
Learning Longer-term Dependencies in RNNs with Auxiliary Losses
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a reconstruction auxiliary loss with truncated supervised BPTT | LSTM · CIFAR10 | test accuracy (%): 49.0 -> 65.9 | Helped |
| Add a reconstruction auxiliary loss with truncated supervised BPTT | LSTM · pMNIST | test accuracy (%): 88.8 -> 92.8 | Helped |
| Add a prediction auxiliary loss with truncated supervised BPTT | LSTM · CIFAR10 | test accuracy (%): 49.0 -> 64.7 | Helped |
| Increase reconstruction sampling from one to 20 segments, using shorter segments (ablation) | r-LSTM · CIFAR10, supervised BPTT length 1 | test accuracy (%): 46.0 -> 48.0 | Helped |
| Increase reconstruction sampling from one to 20 segments, using shorter segments (ablation) | r-LSTM · CIFAR10, supervised BPTT length 0 | test accuracy (%): 37.4 -> 41.6 | Helped |
| Remove the auxiliary loss (ablation) | r-LSTM Truncate300 · CIFAR10 | test accuracy (%): 65.9 -> 49.0 | Hurt |
| Remove joint training of the supervised and auxiliary losses (ablation) | r-LSTM · CIFAR10 | test accuracy (%): +3.6% loss | Hurt |
| Remove Scheduled Sampling for the auxiliary LSTM (ablation) | r-LSTM · CIFAR10 | test accuracy (%) | Hurt |
| Use pretraining without joint training (ablation) | r-LSTM · CIFAR10 | test accuracy (%) | Hurt |
| Sample reconstruction segments from distant past instead of only immediate past (ablation) | r-LSTM · CIFAR10 | test accuracy (%): almost 2% | Helped |
| Allow part of the sampled reconstruction segment to extend into the anchor's future (ablation) | r-LSTM · CIFAR10 | test accuracy (%) | Helped |
| Embed input pixels to the LSTM hidden size (ablation) | r-LSTM · CIFAR10 | test accuracy (%) | Helped |
Closest papers by meaning
- Memory-Efficient Backpropagation Through Time (NeurIPS 2016 · similarity 0.89)
- Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks (ICLR 2018 · similarity 0.88)
- Temporal Pyramid Recurrent Neural Network (AAAI 2020 · similarity 0.87)
- Parallelizing Linear Recurrent Neural Nets Over Sequence Length (ICLR 2018 · similarity 0.87)
- RNNs Incrementally Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients? (ICLR 2020 · similarity 0.87)
- UnICORNN: A recurrent model for learning very long time dependencies (ICML 2021 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.