Research map / Architectures and efficiency / Attention and transformers
Hierarchically Gated Recurrent Neural Network for Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a forget gate to HGRN (vs removing it) (ablation) | HGRN · The Pile 1B | perplexity: 57.42 -> 4.14 | Helped |
| add a forget gate to LRU (ablation) | LRU · The Pile 1B | perplexity: 5.07 -> 4.92 | Helped |
| make forget gate data-dependent rather than data-independent lower bound only (ablation) | HGRN · The Pile 1B | perplexity: 4.84 -> 4.14 | Helped |
| add the monotonically increasing layer-wise lower bound on forget gate (ablation) | HGRN · WikiText-103 | perplexity: 24.71 -> 24.14 | Helped |
| use monotonically increasing lower bound instead of random or decreasing bounds (ablation) | HGRN · WikiText-103 | perplexity: 24.60 -> 24.14 | Helped |
| add the lower bound on forget gate (ablation) | HGRN · Long Range Arena | average accuracy: 51.53 -> 86.91 | Helped |
| add input gate (tie 1-lambda_t) (ablation) | HGRN · WikiText-103 | perplexity: 25.03 -> 24.14 | Helped |
| add output gate (ablation) | HGRN · WikiText-103 | perplexity: 25.50 -> 24.14 | Helped |
| add complex-valued recurrence (ablation) | HGRN · WikiText-103 | perplexity: 25.34 -> 24.14 | Helped |
| make phase theta data-dependent (ablation) | HGRN · WikiText-103 | perplexity: 24.14 -> 28.74 | Hurt |
| HGRN vs Transformer at 1B scale | Transformer · The Pile 1B | perplexity: 4.56 -> 4.14 | Helped |
Closest papers by meaning
- A recurrent neural network without chaos (ICLR 2017 · similarity 0.89)
- Learning Hierarchical Structures with Differentiable Nondeterministic Stacks (ICLR 2022 · similarity 0.89)
- Parallelizing Linear Recurrent Neural Nets Over Sequence Length (ICLR 2018 · similarity 0.89)
- Focused Hierarchical RNNs for Conditional Sequence Processing (ICML 2018 · similarity 0.89)
- Memory-Gated Recurrent Networks (AAAI 2021 · similarity 0.88)
- FlashRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Models (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.