Research map / Architectures and efficiency / Attention and transformers
MoM: Linear Sequence Modeling with Mixture-of-Memories
Jusen Du, Weigao Sun, Disen Lan, Jiaxi Hu, Tao Zhang, Yu Cheng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use mixture of memories instead of single memory in Gated DeltaNet | Gated DeltaNet · Recall-intensive avg (380M) | average score: 24.78 -> 28.16 | Improved |
| use mixture of memories instead of single memory in Gated DeltaNet | Gated DeltaNet · Recall-intensive avg (1.3B) | average score: 32.30 -> 36.04 | Improved |
| use MoM instead of Transformer++ on recall tasks | Transformer++ · Recall-intensive avg (1.3B) | average score: 37.31 -> 36.04 | Hurt |
| use MoM instead of Transformer++ at 380M | Transformer++ · Recall-intensive avg (380M) | average score: 31.70 -> 28.16 | Hurt |
| separate memories instead of expanding a single memory (ablation) | Gated DeltaNet · Recall-intensive avg (380M) | average score: 26.32 -> 28.16 | Improved |
| separate memories instead of expanding a single memory (ablation) | GLA · Recall-intensive avg (380M) | average score: 22.87 -> 23.53 | Improved |
| equalize activated parameters (400M) for MoM vs Gated DeltaNet | Gated DeltaNet · Recall-intensive avg (380M) | average score: 24.78 -> 26.51 | Improved |
| equalize activated parameters on commonsense | Gated DeltaNet · Commonsense avg (400M activated) | average accuracy: 41.44 -> 41.50 | No clear effect |
| MoM on commonsense at 380M vs Gated DeltaNet | Gated DeltaNet · Commonsense avg (380M) | average accuracy: 41.44 -> 41.97 | Improved |
| add 3 Transformer layers to MoM (hybrid) (ablation) | MoM · Recall-intensive avg (380M) | average score: 28.16 -> 37.80 | Improved |
| add 3 Transformer layers to MoM (hybrid) (ablation) | MoM · Commonsense avg (380M) | average accuracy: 41.97 -> 42.38 | Improved |
| increase number of memories from 1 to 8 at activation ratio 0.5 (ablation) | MoM · recall and commonsense benchmarks | average | Improved |
Closest papers by meaning
- Raven: High-Recall Sequence Modeling via Sparse Memory Routing (NeurIPS 2026 · similarity 0.88)
- Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.87)
- Quantifying Memory Utilization with Effective State-Size (ICML 2025 · similarity 0.86)
- RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State (NeurIPS 2026 · similarity 0.86)
- Memorization Sinks: Isolating Memorization during LLM Training (ICML 2025 · similarity 0.86)
- Proteus: Incremental Memory Activation for Long-Context Sequence Modeling (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
