Research map / Architectures and efficiency / RNNs and normalization
Training Recurrent Neural Networks via Forward Propagation Through Time
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train LSTM with FPTT instead of BPTT | 1-layer LSTM · PTB-300 word LM | test perplexity: 117.09 -> 106.27 (about 11 points lower) | Helped |
| train with FPTT instead of BPTT | 3-layer AWD-LSTM-MoS · PTB-w word LM | test perplexity: 51.9 -> 50.96 | Helped |
| train with FPTT instead of BPTT | 3-layer AWD-LSTM · PTB-c character LM | bits per character: 1.183 -> 1.165 | Helped |
| train with FPTT instead of BPTT | 1-layer LSTM · CIFAR-10 sequential | accuracy: 60.11% -> 71.03% | Helped |
| train with FPTT instead of BPTT | 1-layer LSTM · Seq-MNIST | accuracy: 97.71% -> 98.67% | Helped |
| train with FPTT instead of BPTT | 1-layer LSTM · Permute-MNIST | accuracy: 88.91% -> 94.75% | Helped |
| train GRU with FPTT instead of BPTT (ablation) | GRU · CIFAR-10 sequential | accuracy: 66.28% -> 71.37% | Helped |
| train AntisymmetricRNN with FPTT instead of BPTT (ablation) | AntisymmetricRNN · CIFAR-10 sequential | accuracy: 62.41% -> 72.13% | Helped |
| add FPTT-style auxiliary terminal losses to BPTT (ablation) | 1-layer LSTM with BPTT · CIFAR-10 sequential | accuracy: 60.11% -> 65.65%, still below FPTT LSTM 71.03% | Helped |
| increase update frequency K (FPTT-K) toward per-timestep updates (ablation) | LSTM on Add Task T=200 · Add Task (T=200) | MSE convergence | Helped |
| very small alpha (regularizer weight) in FPTT (ablation) | FPTT LSTM · PTB-300 | perplexity | Mixed |
Closest papers by meaning
- Training Recurrent Neural Networks Online by Learning Explicit State Variables (ICLR 2020 · similarity 0.91)
- Memory-Efficient Backpropagation Through Time (NeurIPS 2016 · similarity 0.90)
- Reviving and Improving Recurrent Back-Propagation (ICML 2018 · similarity 0.90)
- Updates of Equilibrium Prop Match Gradients of Backprop Through Time in an RNN with Static Input (NeurIPS 2019 · similarity 0.89)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.87)
- Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.