Research map / Optimization / Optimizers
MDN: Parallelizing Stepwise Momentum for Delta Linear Attention
Yulong Huang, Xiang Liu, Hongxiang Huang, Xiaopeng LIN, Zunchang Liu, Xiaowen Chu, Zeke Xie, Bojun Cheng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add stepwise momentum (ablation) | MDN 400M · LAMBADA | perplexity: 47.01 -> 41.62 | Improved |
| add stepwise momentum (ablation) | MDN 400M · In-context retrieval avg | accuracy: 20.12 -> 26.76 | Improved |
| add output correction (ablation) | MDN 400M · In-context retrieval avg | accuracy: 25.52 -> 26.76 | Improved |
| add lower clamp on log mu (ablation) | MDN 400M · LAMBADA | training stability: NaN -> 41.62 | Improved |
| add alpha_max constraint (ablation) | MDN 400M · LAMBADA | training stability: NaN -> 41.62 | Improved |
| add beta_max constraint (ablation) | MDN 400M · LAMBADA | perplexity: 42.72 -> 41.62 | Improved |
| use eta=tanh+1 instead of 2 sigmoid (ablation) | MDN 400M · LAMBADA | perplexity: 49.10 -> 41.62 | Improved |
| set min log mu to -2 rather than -1 (ablation) | MDN 400M · In-context retrieval avg | accuracy: 25.85 -> 26.76 | Improved |
| set min log mu to -1 rather than -2 (ablation) | MDN 400M · Commonsense reasoning avg | accuracy: 49.42 -> 49.88 | Mixed |
| add stepwise momentum to delta-rule linear attention vs GDN | MDN 1.3B · Needle-in-a-haystack 8k multi-key | accuracy: outperforms strongest baseline by 13.40 | Improved |
| MDN vs GDN | MDN 1.3B · LongBench 16K avg | score: 19.28 -> 20.18 | Improved |
| use MDN instead of Transformer | MDN 1.3B · In-context retrieval avg | accuracy: 40.60 vs 36.14 | Hurt |
Closest papers by meaning
- Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks (NeurIPS 2025 · similarity 0.87)
- Parallelizing Linear Transformers with the Delta Rule over Sequence Length (NeurIPS 2024 · similarity 0.86)
- MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization (NeurIPS 2025 · similarity 0.86)
- Gated Delta Networks: Improving Mamba2 with Delta Rule (ICLR 2025 · similarity 0.86)
- ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-Order Optimization (AAAI 2024 · similarity 0.86)
- Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
