Research map / Architectures and efficiency / RNNs and normalization
Wider and Deeper, Cheaper and Faster: Tensorized LSTMs for Sequence Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add memory cell convolution to 2D tLSTM (ablation) | tLSTM · Wikipedia language modeling | BPC | Helped |
| Add feedback connections (ablation) | 2D tLSTM · Wikipedia language modeling and algorithmic tasks | performance | Helped |
| Add feedback connections (ablation) | 2D tLSTM · Sequential MNIST and sequential pMNIST | accuracy | No clear effect |
| Tensorize 2D tLSTM into 3D (ablation) | tLSTM · Addition and memorization | accuracy | Helped |
| Add channel normalization instead of no normalization (ablation) | 3D tLSTM · Wikipedia language modeling | BPC | Helped |
| Add layer normalization (ablation) | 3D tLSTM · Wikipedia language modeling | BPC | Mixed |
| Replace layer normalization with channel normalization (ablation) | 3D tLSTM · Sequential MNIST and sequential pMNIST | accuracy | Helped |
| Increase depth (ablation) | sLSTM and 2D tLSTM without memory cell convolution · Addition and memorization | accuracy | Hurt |
| Increase tensor size and depth while holding parameter count fixed (ablation) | tLSTM · Sequence learning tasks | performance | Helped |
| Increase depth (ablation) | 2D tLSTM · Sequential MNIST and sequential pMNIST | accuracy | Helped |
| Increase depth (ablation) | 2D tLSTM without memory cell convolution · Sequential MNIST and sequential pMNIST | accuracy | No clear effect |
Closest papers by meaning
- Kernel-Based Approaches for Sequence Modeling: Connections to Neural Methods (NeurIPS 2019 · similarity 0.89)
- Tensor-Train Recurrent Neural Networks for Video Classification (ICML 2017 · similarity 0.89)
- xLSTM: Extended Long Short-Term Memory (NeurIPS 2024 · similarity 0.88)
- Towards Binary-Valued Gates for Robust LSTM Training (ICML 2018 · similarity 0.88)
- Convolutional Sequence to Sequence Learning (ICML 2017 · similarity 0.88)
- Parallelizing non-linear sequential models over the sequence length (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.