Research map / Architectures and efficiency / RNNs and normalization
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add zoneout to an LSTM | Char-PTB · Char-PTB test | BPC: 1.356 -> 1.252 | Helped |
| Add zoneout to an LSTM | Word-PTB · Word-PTB test | perplexity: 78.4 to 77.4 | Helped |
| Add zoneout to an LSTM | Text8 · Text8 test | BPC: 1.408 -> 1.336 | Helped |
| Add zoneout to an LSTM | pMNIST · pMNIST test | error rate: 0.102 -> 0.069 | Helped |
| Add zoneout to recurrent batch normalization | LSTM on pMNIST · pMNIST test | error rate: 0.046 -> 0.041 | Helped |
| Add zoneout to recurrent connections | LSTM on pMNIST · pMNIST test | error rate: 0.075 -> 0.069 | Helped |
| Use separate masks sampled per unit instead of a shared cell-and-hidden mask (ablation) | LSTM on Char-PTB · Char-PTB | BPC | Helped |
| Set the cell zoneout probability to 0.5 while keeping hidden-state probability at 0.05 (ablation) | LSTM on Char-PTB · Char-PTB | BPC: 1.27 BPC | Helped |
| Use zoneout on cells and hidden states with probabilities 0.15 (ablation) | LSTM on pMNIST · pMNIST validation | error rate: 0.092 -> 0.063 | Helped |
| Replace stochastic training masks with static identity connections (ablation) | LSTM on pMNIST · pMNIST | error rate | Hurt |
| Add zoneout to the word-level LSTM without non-recurrent dropout (ablation) | Word-PTB · Word-PTB | perplexity | Hurt |
| Use stochastic depth instead of per-unit zoneout (ablation) | LSTM on Char-PTB · Char-PTB test | BPC: 1.252 vs 1.343 | Hurt |
Closest papers by meaning
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.87)
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.87)
- GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling (NeurIPS 2025 · similarity 0.87)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.86)
- Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws (NeurIPS 2026 · similarity 0.86)
- Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.