Research map / Architectures and efficiency / Attention and transformers
Mogrifier LSTM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use mutual, alternating input-state gating before the LSTM update | LSTM · PTB EN test, no dyneval | perplexity: 3.99985 -> 3.93124 | Helped |
| Use mutual, alternating input-state gating before the LSTM update | LSTM · Wikitext-2 EN test, no dyneval | perplexity: 4.09557 -> 4.03665 | Helped |
| Use mutual, alternating input-state gating before the LSTM update | LSTM · MWC EN test, no dyneval | bits per character: 0.92752 -> 0.90439 | Helped |
| Use mutual, alternating input-state gating before the LSTM update | LSTM · MWC FI test, no dyneval | bits per character: 0.94772 -> 0.91896 | Helped |
| Use mutual, alternating input-state gating before the LSTM update | LSTM · Enwik8 EN test, dyneval, 96M | bits per character: 0.70699 -> 0.68516 | Helped |
| Use multiple rounds of mutual gating rather than no gating or fewer rounds (ablation) | Mogrifier LSTM · PTB EN | perplexity | Helped |
| Use low-rank factorization of the gating matrices (ablation) | Mogrifier LSTM · PTB EN | perplexity | Helped |
| Replace alternating gating with gates conditioned only on the original inputs (ablation) | Mogrifier LSTM · PTB EN | per-word validation cross-entropy | Hurt |
| Add a learnable scaling factor to the LSTM inputs without data-dependent gating (ablation) | LSTM · Language modelling | performance | No clear effect |
| Reduce Mogrifier gating rank to less than 5 (ablation) | Mogrifier LSTM · Language modelling | performance | Hurt |
| Factorize the LSTM gate matrices (ablation) | LSTM · Language modelling | performance | No clear effect |
| Use multiplicative LSTM instead of the Mogrifier (ablation) | LSTM · PTB | performance | No clear effect |
Closest papers by meaning
- On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models (NeurIPS 2026 · similarity 0.88)
- xLSTM: Extended Long Short-Term Memory (NeurIPS 2024 · similarity 0.88)
- L$^2$M: Mutual Information Scaling Law for Long-Context Language Modeling (NeurIPS 2025 · similarity 0.88)
- Liger: Linearizing Large Language Models to Gated Recurrent Structures (ICML 2025 · similarity 0.87)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.87)
- Language Models Are Implicitly Continuous (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.