Research map / Architectures and efficiency / RNNs and normalization
Fraternal Dropout
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add fraternal dropout to AWD-LSTM 3-layer | AWD-LSTM 3-layer · Penn Treebank word-level | perplexity: 60.0 -> 58.9 (validation); 57.3 -> 56.8 (test) | Helped |
| add fraternal dropout to AWD-LSTM 3-layer | AWD-LSTM 3-layer · WikiText-2 word-level | perplexity: 68.6 -> 66.8 (validation); 65.8 -> 64.1 (test) | Helped |
| add fraternal dropout with κ = 0.005 | Show and Tell baseline · Microsoft COCO | BLEU-4: 25.6 -> 26.3 | Helped |
| use fraternal dropout instead of traditional dropout + Π-model (ablation) | ResNet-56 · CIFAR-10 semi-supervised | accuracy (test): 78.5 -> 79.1 | Helped |
| use fraternal dropout instead of traditional dropout (ablation) | ResNet-56 · CIFAR-10 without unlabeled data | accuracy (test): 76.9 -> 77.6 | Helped |
| add target loss to both networks rather than only one (ablation) | Π-model-style LSTM training · Penn Treebank word-level | convergence speed | Helped |
| apply target loss to both copies in ELD (ablation) | ELD (ELDM) · Penn Treebank word-level, single-layer LSTM | generalization | Hurt |
| replace ELD with fraternal dropout (ablation) | AWD-LSTM 3-layer · Penn Treebank validation | perplexity: 59.4 vs 59.4 | No clear effect |
| use fraternal dropout rather than ELD (ablation) | AWD-LSTM 3-layer · Penn Treebank validation, top-performing hyperparameter runs | perplexity: 60.1 -> 59.6 (Top5 avg); 60.5 -> 59.9 (Top10 avg) | Helped |
| add one NT-ASGD fine-tuning step (ablation) | Fraternal dropout + AWD-LSTM 3-layer · Penn Treebank word-level | perplexity: 59.8 -> 58.9 (validation); 58.0 -> 56.8 (test) | Helped |
| use two fine-tuning steps instead of one (ablation) | Fraternal dropout + AWD-LSTM 3-layer · Penn Treebank word-level | perplexity: 58.9 -> 58.5 (validation); 56.8 -> 56.2 (test) | Helped |
| replace averaged-mask evaluation with Monte Carlo mask sampling (ablation) | AWD-LSTM 3-layer · Penn Treebank word-level | perplexity: 60.7 -> 64.4 (MC50 validation); 58.8 -> 62.1 (MC50 test) | Hurt |
Closest papers by meaning
- Adversarial Dropout for Recurrent Neural Networks (AAAI 2019 · similarity 0.93)
- Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for Efficient Training (NeurIPS 2021 · similarity 0.92)
- A Theoretically Grounded Application of Dropout in Recurrent Neural Networks (NeurIPS 2016 · similarity 0.90)
- R-Drop: Regularized Dropout for Neural Networks (NeurIPS 2021 · similarity 0.89)
- Dropout with Expectation-linear Regularization (ICLR 2017 · similarity 0.88)
- Jumpout : Improved Dropout for Deep Neural Networks with ReLUs (ICML 2019 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.