Research map / Architectures and efficiency / RNNs and normalization
A Tensor Decomposition Perspective on Second-order RNNs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies second-order RNNs whose weight tensor is CP-decomposed (CPRNN), proving rank-hidden size expressivity relations to RNNs, 2RNNs and MIRNNs, and validates them with character-level Penn Treebank experiments.
Problem
Second-order RNNs (2RNNs) are more expressive than RNNs but their third-order weight tensor is too large to be tractable; prior CP-compressed variants lacked theoretical analysis of how rank and hidden size control capacity.
Approach
Parameterizes the 2RNN recurrent tensor with a CP decomposition of rank R, so R interpolates between RNNs (R=0) and 2RNNs; proves inclusion, saturation and strict-inclusion results in rank and hidden size, and shows MIRNNs are CPRNNs with rank equal to hidden size and a diagonal factor.
Key results
- CPRNN with the right rank and hidden size outperforms RNNs, 2RNNs and MIRNNs at every fixed parameter budget tested on Penn Treebank (BPC, averaged over 10 seeds).
- With a CPRNN of rank 4 at hidden size 1024 (about 2% more parameters), BPC closes 59% of the relative gap between RNN and 2RNN.
- Theory: CPRNN rank increases expressivity strictly up to the maximal typical CP rank and saturates at the maximal CP rank, where it equals 2RNN; MIRNN is a strict subset of CPRNN for rank above hidden size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a CPRNN with tuned rank and hidden size at a fixed parameter budget instead of an RNN, 2RNN, or MIRNN | Penn Treebank character-level language modelling · Penn Treebank test | BPC | Helped |
| Increase CPRNN rank | CPRNN · Penn Treebank test | BPC | Helped |
| Increase hidden size | RNN, MIRNN, 2RNN, and CPRNN · Penn Treebank test | BPC | Helped |
| Use CPRNN rank 4 instead of an RNN at hidden size 1024 | CPRNN · Penn Treebank test | relative RNN–2RNN BPC gap closed: 59% | Helped |
| Remove first-order terms (CPBIRNN instead of CPRNN) (ablation) | CPRNN · Penn Treebank test, fixed parameter count | BPC | Hurt |
| Use very low rank or very high rank, which leaves hidden size too small (ablation) | CPBIRNN · Penn Treebank test, fixed parameter counts of 111K and 1.7M | BPC | Hurt |
Surprising findings
- At fixed parameter count, rank and hidden size trade off, producing two underfitting regimes for CPBIRNNs (rank too low or hidden size too low).
Limitations
- Strict inclusion proven for linear activations and CPBIRNN only for analytic invertible activations; conjectured for tanh and ReLU.
- Gap between typical and maximal CP rank left open, so the saturation point is not tight.
- Experiments limited to character-level Penn Treebank with a single hidden-size grid.
Experimental setup
Datasets: Penn Treebank
Benchmarks: Penn Treebank
Metrics: bits-per-character
Baselines: RNN MIRNN 2RNN CPBIRNN
Builds on: second-order RNN MIRNN CP decomposition weighted finite automata
Closest papers by meaning
- LightRNN: Memory and Computation-Efficient Recurrent Neural Networks (NeurIPS 2016 · similarity 0.89)
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.88)
- Tensor-Train Recurrent Neural Networks for Video Classification (ICML 2017 · similarity 0.88)
- RNNs implicitly implement tensor-product representations (ICLR 2019 · similarity 0.88)
- FlashRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Models (ICLR 2026 · similarity 0.88)
- Fast-Slow Recurrent Neural Networks (NeurIPS 2017 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.