Research map / Time series / Long-term forecasting
Fast Saturating Gate for Learning Long Time Scales with Recurrent Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the sigmoid forget gate with the fast gate | LSTM · sMNIST test | accuracy: 98.88 $\pm$ 0.12 -> 99.05 $\pm$ 0.04 | Helped |
| Replace the sigmoid forget gate with the fast gate | LSTM · psMNIST test | accuracy: 95.71 $\pm$ 0.02 -> 96.18 $\pm$ 0.14 | Helped |
| Replace the sigmoid forget gate with the fast gate | LSTM · sCIFAR test | accuracy: 69.14 $\pm$ 0.39 -> 70.06 $\pm$ 0.38 | Helped |
| Use the fast gate instead of the sigmoid gate | LSTM language model · Penn Treebank all tokens | perplexity: 60.22 -> 60.09 | Helped |
| Use the fast gate instead of the sigmoid gate | LSTM language model · Penn Treebank tokens 1K-10K | perplexity: 27.72 -> 27.70 | Helped |
| Use the fast gate instead of the sigmoid gate | LSTM language model · Penn Treebank tokens 100-1K | perplexity: 170.25 -> 166.68 | Hurt |
| Use the fast gate instead of the sigmoid gate | LSTM language model · Penn Treebank tokens < 100 | perplexity: 2026.87 -> 1975.51 | Hurt |
| Replace sigmoid gate with normalized softsign | LSTM · sMNIST test | accuracy: 98.88 $\pm$ 0.12 -> 97.50 $\pm$ 0.58 | Hurt |
| Replace sigmoid gate with refine gate | LSTM · psMNIST processing | time per epoch: 14.3 min. -> 22.7 min. | Hurt |
| Add a second sinh composition to the fast gate (ablation) | LSTM · sMNIST test | accuracy: 99.05 $\pm$ 0.04 -> 99.02 $\pm$ 0.07 | No clear effect |
| Add a second sinh composition to the fast gate (ablation) | LSTM · psMNIST test | accuracy: 96.18 $\pm$ 0.14 -> 96.22 $\pm$ 0.08 | No clear effect |
| Add a second sinh composition to the fast gate (ablation) | LSTM · sCIFAR test | accuracy: 70.06 $\pm$ 0.38 -> 69.66 $\pm$ 0.41 | Hurt |
Closest papers by meaning
- Can recurrent neural networks warp time? (ICLR 2018 · similarity 0.88)
- Improving the Gating Mechanism of Recurrent Neural Networks (ICML 2020 · similarity 0.88)
- Structured Sparsification of Gated Recurrent Neural Networks (AAAI 2020 · similarity 0.86)
- Memory-Gated Recurrent Networks (AAAI 2021 · similarity 0.86)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.83)
- Exploring the Promise and Limits of Real-Time Recurrent Learning (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.