Research map / Architectures and efficiency / Mixture of experts
UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning
Zihao Huang, Yu Bao, Qiyang Min, Siyan Chen, Ran Guo, Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, zhou Xun, Siyuan Qiao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PEER-style FFN values instead of value embeddings (ablation) | UltraMemV2 · UltraMemV2-430M/5B | training loss and Open-Benchmark accuracy | Improved |
| single value projector and separate queries per Tucker rank (ablation) | UltraMemV2 · UltraMemV2-430M/5B after 1.5T tokens | Open-Benchmark accuracy: +0.7 points; loss -0.0026 | Improved |
| use one head instead of two (ablation) | UltraMemV2 · UltraMemV2-430M/5B after 1T tokens | Open-Benchmark accuracy: +0.2 points; loss -8e-4 | No clear effect |
| remove Tucker core penalty loss (ablation) | UltraMemV2 · UltraMemV2-430M/5B after 800B tokens | Open-Benchmark accuracy: +0.3 points | No clear effect |
| add balance loss (beta=0.001) (ablation) | UltraMemV2 · UltraMemV2-430M/5B | training loss and accuracy: helps at TopM=47, hurts at TopM=94 | Mixed |
| increase number of UltraMemV2 layers (2, 5, 10, 20) (ablation) | UltraMemV2 · UltraMemV2-430M/5B | Open-Benchmark accuracy: +2.3, +0.9, +1.1 | Improved |
| put a memory layer in every transformer block (ablation) | UltraMemV2 · UltraMemV2 | gain from continued training: +5.7 (4 layers) -> +8.3 (every block, 1.25B/12.5B) | Improved |
| raise activated values per layer from top256 to top768 (ablation) | UltraMemV2 · UltraMemV2-2.5B | Openbench All / Hardbench All (after 250B CT): 68.3 -> 69.1; 27.9 -> 30.0 | Improved |
| use Ring instead of NoRing value sharing (S9) (ablation) | UltraMemV2 · UltraMemV2-500M/6.8B | Open-Benchmark accuracy: about +1 point | Improved |
| share value tables across layers (S9-Ring) (ablation) | UltraMemV2 · UltraMemV2-500M/6.8B | Open-Benchmark accuracy: +1 point; S16-Ring no further gain | Improved |
| constant 1x value learning rate instead of 4x decaying to 1x (ablation) | UltraMemV2 · UltraMemV2-500M/6.8B at 1.4T tokens | Open-Benchmark accuracy: +0.4 points at 1.4T; behind early | Mixed |
| use UltraMemV2 instead of SeedMoE-2.5B/30B | SeedMoE · 32K long-context | In-context learning: 21.6 -> 29.5 | Improved |
Closest papers by meaning
- Ultra-Sparse Memory Network (ICLR 2025 · similarity 0.92)
- Memory Layers at Scale (ICML 2025 · similarity 0.88)
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.87)
- Oracle-MoE: Locality-preserving Routing in the Oracle Space for Memory-constrained Large Language Model Inference (ICML 2025 · similarity 0.86)
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.86)
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
