Research map / Architectures and efficiency / RNNs and normalization
Kronecker Recurrent Units
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KRU instead of FC uRNN (N=512) | Pixel-by-pixel recurrent model · Permuted MNIST | test accuracy: 94.1 -> 94.5 | Helped |
| Use KRU instead of LSTM | Pixel-by-pixel recurrent model · Unpermuted MNIST | test accuracy: 97.8 -> 96.4 | Hurt |
| Use KRU instead of RNN | Character-level Penn TreeBank model · Penn TreeBank character-level | test BPC: 1.60 -> 1.60 | No clear effect |
| Use KRU-LSTM instead of LSTM | Character-level Penn TreeBank model · Penn TreeBank character-level | test BPC: 1.45 -> 1.47 | Hurt |
| Use KRU-LSTM instead of LSTM | Polyphonic music model · JSB Chorales | test average negative log-likelihood: 8.67 -> 8.54 | Helped |
| Use KRU-LSTM instead of RNN | Polyphonic music model · Piano-midi | test average negative log-likelihood: 9.03 -> 8.18 | Helped |
| Increase the soft-unitary regularization amplitude (ablation) | KRU · JSB Chorales validation | validation performance | Mixed |
| Increase the soft-unitary regularization amplitude (ablation) | KRU · Piano-midi validation | validation performance | Mixed |
| Increase the soft-unitary regularization amplitude beyond 1e-5 (ablation) | KRU · TIMIT phoneme recognition validation | validation error | Hurt |
| Use a unitary-initialized, fixed recurrent matrix instead of learning the recurrent matrix (ablation) | KRU · Copy memory, T=1000 and T=2000 | average cross entropy | Helped |
| Use strict unitary recurrent models rather than soft-unitary KRU | Recurrent models · Adding problem | learning stability | Hurt |
Closest papers by meaning
- Approximating Real-Time Recurrent Learning with Random Kronecker Factors (NeurIPS 2018 · similarity 0.88)
- Optimal Kronecker-Sum Approximation of Real Time Recurrent Learning (ICML 2019 · similarity 0.88)
- Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling (AAAI 2018 · similarity 0.87)
- Complex Unitary Recurrent Neural Networks Using Scaled Cayley Transform (AAAI 2019 · similarity 0.87)
- Full-Capacity Unitary Recurrent Neural Networks (NeurIPS 2016 · similarity 0.87)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.