Research map / Architectures and efficiency / RNNs and normalization
Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ordered-neuron gating with cumax master gates | 3-layer tied LSTM language model · Penn Treebank test | perplexity: 57.3 -> 56.17 | Helped |
| Use the second-layer gate sequence for parse induction | ON-LSTM · WSJ | Parsing F1: 37.4 (0.3) -> 47.7 (1.5) | Helped |
| Use the second-layer gate sequence rather than the first-layer sequence | ON-LSTM · WSJ | Parsing F1: 20.0 (2.8) -> 47.7 (1.5) | Helped |
| Use the second-layer gate sequence rather than the third-layer sequence | ON-LSTM · WSJ | Parsing F1: 36.6 (3.3) -> 47.7 (1.5) | Helped |
| Add ordered-neuron recurrence | LSTM language model · Targeted syntactic evaluation, long-term dependencies | accuracy: 0.60 -> 0.66; 0.52 -> 0.57; 0.51 -> 0.54; 0.74 -> 0.74; 0.68 -> 0.67; 0.58 -> 0.57 | Mixed |
| Add ordered-neuron recurrence | LSTM language model · Targeted syntactic evaluation, short-term dependencies | accuracy: 1.00 -> 0.99; 0.98 -> 0.95; 0.92 -> 0.89; 0.88 -> 0.84; 0.81 -> 0.78 | Mixed |
| Add ordered-neuron recurrence | LSTM language model · Targeted syntactic evaluation, reflexive anaphora | accuracy: 0.82 -> 0.89; 0.80 -> 0.86; 0.58 -> 0.57 | Mixed |
| Add ordered-neuron recurrence | LSTM language model · Targeted syntactic evaluation, negative polarity items | accuracy: 1.00 -> 0.18; 0.01 -> 0.50; 0.63 -> 0.07; 0.95 -> 0.59; 0.00 -> 0.20; 0.04 -> 0.11 | Mixed |
Closest papers by meaning
- Ordered Memory (NeurIPS 2019 · similarity 0.89)
- Memory Architectures in Recurrent Neural Network Language Models (ICLR 2018 · similarity 0.87)
- Sentence Ordering and Coherence Modeling using Recurrent Neural Networks (AAAI 2018 · similarity 0.87)
- Mogrifier LSTM (ICLR 2020 · similarity 0.86)
- Learning Hierarchical Structures with Differentiable Nondeterministic Stacks (ICLR 2022 · similarity 0.86)
- Order-Independence Without Fine Tuning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.