Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| variational (same-mask across time steps) dropout on recurrent layers, embeddings, inputs and outputs instead of naive input/output dropout | LSTM language model (large) · Penn Treebank | test perplexity: 78.4 -> 73.4 | Helped |
| variational dropout instead of naive dropout | LSTM language model (medium) · Penn Treebank | test perplexity: 97.0 -> 78.6 (MC) | Helped |
| weight tying vs untied weights under dropout approximation (ablation) | Variational LSTM · Penn Treebank | test perplexity: tied 74.1 vs untied 73.4 with MC (large); tied 75.0 vs untied 75.2 with approximation (large) | Mixed |
| adding embedding dropout to Gal-style recurrent-mask baseline (Merity-style variant) (ablation) | LSTM with same-mask cell dropout baseline · Penn Treebank (large) | validation perplexity: 122.9 -> 88.8 | Helped |
| no early stopping (large model with naive or cell-mask baseline without embedding dropout) (ablation) | LSTM with cell-mask dropout baseline · Penn Treebank (large) | validation perplexity: validation rises to 131 within 30 epochs | Hurt |
| ensembling 10 Variational LSTMs | Variational LSTM · Penn Treebank | test perplexity: 69.5 -> 68.7 | Helped |
| embedding dropout (drop word types) with recurrent-layer dropout p_U=0.5 (ablation) | Variational LSTM on sentiment analysis · Cornell movie reviews | test error behaviour | Helped |
| dropout on recurrent connections (same mask over time) vs none (ablation) | LSTM on sentiment analysis (p_E=0) · Cornell movie reviews | overfitting / test error | No clear effect |
| weight decay removed vs tuned with variational dropout (ablation) | Variational LSTM · Penn Treebank / sentiment | test error / perplexity | Helped |
| dropout approximation (mean propagation) instead of MC dropout at test time (ablation) | Variational GRU / LSTM on sentiment analysis · Cornell movie reviews | test error | No clear effect |
| tied-weights vs untied-weights LSTM parametrisation (training speed) (ablation) | Variational LSTM · Penn Treebank (large) | words per second (WPS): 2.4K vs 1.6K WPS | Helped |
Closest papers by meaning
- Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for Efficient Training (NeurIPS 2021 · similarity 0.91)
- Adversarial Dropout for Recurrent Neural Networks (AAAI 2019 · similarity 0.91)
- Fraternal Dropout (ICLR 2018 · similarity 0.90)
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (ICML 2016 · similarity 0.90)
- Stochastic Modified Equations and Dynamics of Dropout Algorithm (ICLR 2024 · similarity 0.86)
- BARNN: A Bayesian Autoregressive and Recurrent Neural Network (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.