Research map / Architectures and efficiency / RNNs and normalization
Dilated Recurrent Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-resolution dilated recurrent connections instead of regular recurrent connections | Vanilla RNN · Copy memory T=500 and T=1,000 | cross-entropy loss | Helped |
| use dilated Vanilla instead of regular Vanilla | Vanilla RNN · Penn Treebank | BPC | Helped |
| use dilated GRU instead of regular GRU | GRU · Penn Treebank | BPC: 1.33 -> 1.27 | Helped |
| use dilated Vanilla instead of regular skip Vanilla | Vanilla RNN · MNIST unpermuted | test accuracy | Helped |
| use dilated recurrent architecture instead of regular recurrent skips | Vanilla RNN · Noisy MNIST T=1,000 and T=2,000 | test accuracy | Helped |
| use raw-waveform Dilated GRU rather than raw-waveform Fused GRU | GRU · VCTK raw waveform | accuracy: 0.65 -> 0.74 | Helped |
| use dilated GRU rather than GRU | GRU · MNIST unpermuted | test accuracy: 99.1 -> 99.0 / 99.2 | No clear effect |
| increase the number of dilated Vanilla layers (ablation) | Dilated Vanilla · Noisy MNIST T=1,000 | classification accuracy | Helped |
| start dilation above 1 by removing bottom layers (ablation) | Dilated Vanilla · Noisy MNIST T=1,000 | evaluation accuracy | Hurt |
| start dilation at 2 instead of 1 (ablation) | Dilated Vanilla · Noisy MNIST T=1,000 | evaluation accuracy | No clear effect |
| start dilation between 4 and 16 instead of 1 (ablation) | Dilated Vanilla · Noisy MNIST T=1,000 | evaluation accuracy | Hurt |
| start dilation at 64 rather than 1 (ablation) | Dilated Vanilla · Noisy MNIST T=1,000 | training time: within 17 minutes | Helped |
Closest papers by meaning
- Capacity and Trainability in Recurrent Neural Networks (ICLR 2017 · similarity 0.89)
- Independently Recurrent Neural Network (IndRNN): Building a Longer and Deeper RNN (CVPR 2018 · similarity 0.89)
- Parallelizing Linear Recurrent Neural Nets Over Sequence Length (ICLR 2018 · similarity 0.89)
- Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks (ICLR 2018 · similarity 0.89)
- Quasi-Recurrent Neural Networks (ICLR 2017 · similarity 0.88)
- Fast-Slow Recurrent Neural Networks (NeurIPS 2017 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.