Research map / Architectures and efficiency / RNNs and normalization
RNNs implicitly implement tensor-product representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use tree-position roles instead of bag-of-words roles | TPDN approximation of SST sentence encodings · SNLI premise sentence encodings | normalized MSE: 0.27 -> 0.17 | Helped |
| Use tree-position roles instead of bag-of-words roles | TPDN approximation of SPINN · SNLI | prediction agreement: 0.58 -> 0.76 | Helped |
| Use tree-position roles instead of bag-of-words roles | TPDN approximation of SST · SNLI | prediction agreement: 0.49 -> 0.67 | Helped |
| Use bag-of-words roles instead of structural roles | TPDN approximation of sentence encoders · SST, MRPC, and STS-B | prediction agreement | No clear effect |
| Train the decoder to reverse rather than autoencode sequences | Unidirectional GRU encoder/decoder · Digit sequences | substitution accuracy: 0.871 -> 0.062 (LTR); 0.112 -> 0.986 (RTL) | Mixed |
| Train the decoder to sort rather than autoencode sequences | Unidirectional GRU encoder/decoder · Digit sequences | substitution accuracy: 0.001 -> 0.892 (BOW); 0.992 -> 0.923 (Bi) | Mixed |
| Train the decoder to interleave rather than autoencode sequences | Unidirectional GRU encoder/decoder · Digit sequences | substitution accuracy: 0.871 -> 0.269 (LTR); 0.992 -> 0.992 (Bi) | Mixed |
| Use a tree-based decoder rather than a unidirectional decoder | Autoencoder with unidirectional encoder · Digit sequences, autoencoding | substitution accuracy: 0.992 -> 0.881 (Bi to Tree role scheme) | Helped |
| Remove the final linear layer (ablation) | TPDN approximating the reversal model; filler dimension 10, role dimension 6 · Digit sequences, reversal | substitution accuracy: 0.99 -> 0.0002 | Hurt |
| Replace tensor product binding with circular convolution at 20 dimensions (ablation) | TPDN approximation of the reversal model · Digit sequences, reversal; RTL roles | substitution accuracy: 0.993 -> 0.045 | Hurt |
| Replace tensor product binding with circular convolution at 60 dimensions (ablation) | TPDN approximation of the reversal model · Digit sequences, reversal; RTL roles | substitution accuracy: 0.988 -> 0.964 | No clear effect |
| Replace tensor product binding with elementwise product at 20 dimensions (ablation) | TPDN approximation of the reversal model · Digit sequences, reversal; RTL roles | substitution accuracy: 0.993 -> 0.617 | Hurt |
Closest papers by meaning
- Attentive Tensor Product Learning (AAAI 2019 · similarity 0.89)
- Mapping natural-language problems to formal-language solutions using structured neural representations (ICML 2020 · similarity 0.88)
- A Unifying Perspective on Language Model Interpretability (NeurIPS 2026 · similarity 0.88)
- A Tensor Decomposition Perspective on Second-order RNNs (ICML 2024 · similarity 0.88)
- Question-Answering with Grammatically-Interpretable Representations (AAAI 2018 · similarity 0.88)
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.