Research map / Architectures and efficiency / RNNs and normalization
Self-Instantiated Recurrent Units with Dynamic Soft Recursion
Aston Zhang, Yi Tay, Yikang Shen, Alvin Chan, Shuai Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Self-IRU instead of ON-LSTM on longest sequences | logical inference model · Logical inference, 12 operations | accuracy: 76.0 -> 88.0 | Improved |
| Use Self-IRU instead of Tree-LSTM with ground-truth syntax | logical inference model · Logical inference, 12 operations | accuracy: 87.0 -> 88.0 | Improved |
| Use Self-IRU encoder instead of TrellisNet with 8x more parameters | sequence encoder · Pixel-wise CIFAR-10 | accuracy: 73.42 -> 73.01 | No clear effect |
| Use Self-IRU instead of Transformer | sequence encoder · Pixel-wise CIFAR-10 | accuracy: 62.20 -> 73.01 | Improved |
| Replace recurrent decoder of TranX with Self-IRU | TranX · Django code generation | accuracy: 77.2 -> 78.3 | Improved |
| Replace recurrent decoder of TranX with Self-IRU | TranX · Atis semantic parsing | accuracy: 87.7 -> 88.4 | Improved |
| Replace recurrent decoder of TranX with Self-IRU | TranX · Geo semantic parsing | accuracy: 88.6 -> 88.6 | No clear effect |
| Use Self-IRU instead of TCN on JSB Chorales | music model · JSB Chorales | negative log-likelihood: 8.13 -> 8.12 | No clear effect |
| Use Self-IRU instead of TCN on Nottingham | music model · Nottingham | negative log-likelihood: 2.95 -> 2.88 | Improved |
| Increase max recursion depth from 1 to 3 with linear base transformation (ablation) | Self-IRU · Atis semantic parsing | accuracy: 88.40 -> 87.80 | Hurt |
| Use LSTM instead of linear base transformation (depth 1) (ablation) | Self-IRU · Atis semantic parsing | accuracy: 88.40 -> 86.61 | Hurt |
| Use LSTM instead of linear base transformation (depth 1) (ablation) | Self-IRU · Django code generation | accuracy: 77.56 -> 78.33 | Improved |
Closest papers by meaning
- Universal Transformers (ICLR 2019 · similarity 0.87)
- Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (ICML 2018 · similarity 0.86)
- Resurrecting Recurrent Neural Networks for Long Sequences (ICML 2023 · similarity 0.86)
- FlashRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Models (ICLR 2026 · similarity 0.86)
- Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues (ICML 2024 · similarity 0.86)
- Encoding Recurrence into Transformers (ICLR 2023 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
