AI Paper Atlas

Research map / Optimization / Optimizers

MDN: Parallelizing Stepwise Momentum for Delta Linear Attention

Yulong Huang, Xiang Liu, Hongxiang Huang, Xiaopeng LIN, Zunchang Liu, Xiaowen Chu, Zeke Xie, Bojun Cheng

ICML 2026 · Optimization › Optimizers · approach: Memory-efficient LLM optimizers · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add stepwise momentum (ablation)MDN 400M · LAMBADAperplexity: 47.01 -> 41.62Improved
add stepwise momentum (ablation)MDN 400M · In-context retrieval avgaccuracy: 20.12 -> 26.76Improved
add output correction (ablation)MDN 400M · In-context retrieval avgaccuracy: 25.52 -> 26.76Improved
add lower clamp on log mu (ablation)MDN 400M · LAMBADAtraining stability: NaN -> 41.62Improved
add alpha_max constraint (ablation)MDN 400M · LAMBADAtraining stability: NaN -> 41.62Improved
add beta_max constraint (ablation)MDN 400M · LAMBADAperplexity: 42.72 -> 41.62Improved
use eta=tanh+1 instead of 2 sigmoid (ablation)MDN 400M · LAMBADAperplexity: 49.10 -> 41.62Improved
set min log mu to -2 rather than -1 (ablation)MDN 400M · In-context retrieval avgaccuracy: 25.85 -> 26.76Improved
set min log mu to -1 rather than -2 (ablation)MDN 400M · Commonsense reasoning avgaccuracy: 49.42 -> 49.88Mixed
add stepwise momentum to delta-rule linear attention vs GDNMDN 1.3B · Needle-in-a-haystack 8k multi-keyaccuracy: outperforms strongest baseline by 13.40Improved
MDN vs GDNMDN 1.3B · LongBench 16K avgscore: 19.28 -> 20.18Improved
use MDN instead of TransformerMDN 1.3B · In-context retrieval avgaccuracy: 40.60 vs 36.14Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy