Research map / Architectures and efficiency / RNNs and normalization
Multi-timescale Representation Learning in LSTM Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Impose fixed multi-timescale structure, with short timescales in layer 1 and an Inverse Gamma distribution in layer 2 | 3-layer LSTM language model · PTB test, all tokens | perplexity: 61.40 -> 59.69 | Helped |
| Impose fixed multi-timescale structure, with short timescales in layer 1 and an Inverse Gamma distribution in layer 2 | 3-layer LSTM language model · WikiText-2 test, all tokens | perplexity: 69.88 -> 68.08 | Helped |
| Impose fixed multi-timescale structure | 3-layer LSTM language model · PTB test, below 100 | perplexity: 2252.50 -> 2100.89 | Helped |
| Impose fixed multi-timescale structure | 3-layer LSTM language model · WikiText-2 test, below 100 | perplexity: 4631.08 -> 4318.72 | Helped |
| Impose fixed multi-timescale structure | 3-layer LSTM language model · PTB test, above 10K | perplexity: 6.82 -> 6.84; 95% CI [-0.06, 0.02] | No clear effect |
| Impose fixed multi-timescale structure | 3-layer LSTM language model · WikiText-2 test, above 10K | perplexity: 7.49 -> 7.46; 95% CI [0.01,0.06] | Helped |
| Impose fixed Inverse Gamma timescales with α=1.50 | 256-unit LSTM · Dyck-2 test sequences | percent correctly predicted: 91.66% -> 93.82% | Helped |
| Use an Inverse Gamma timescale distribution rather than a narrow Gaussian | LSTM language model trained on PTB · PTB timescale distribution fit | Kolmogorov-Smirnov fit | Helped |
| Use a narrow Gaussian timescale distribution rather than an Inverse Gamma distribution | LSTM language model trained on Markov-PTB · Markov-PTB timescale distribution fit | Kolmogorov-Smirnov fit | Helped |
| Set the Inverse Gamma shape parameter to α=0.56 (ablation) | Multi-timescale LSTM language model · PTB and WikiText-2 | perplexity | Helped |
| Assign short timescales in layer 1 and longer, distributed timescales in layer 2 (ablation) | Multi-timescale LSTM language model · PTB and WikiText-2 word-ablation analysis | cell-state information retention | Helped |
| Ablate long-timescale units (ablation) | Multi-timescale LSTM · PTB test, low-frequency words | perplexity | Hurt |
Closest papers by meaning
- xLSTM: Extended Long Short-Term Memory (NeurIPS 2024 · similarity 0.89)
- On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models (NeurIPS 2026 · similarity 0.88)
- Universal One-third Time Scaling in Learning Peaked Distributions (ICML 2026 · similarity 0.88)
- Arrows of Time for Large Language Models (ICML 2024 · similarity 0.87)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.87)
- Scaling Law with Learning Rate Annealing (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.