Research map / Architectures and efficiency / Attention and transformers
Dynamic Linear Attention
Xin Wang, Hui Shen, Boyuan Zheng, Xueshen Liu, Minkyoung Cho, Zhongwei Wan, Zesen Zhao, Zhuoqing Mao, Shen Yan, Mi Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace log-linear fixed merging with DLA | Mamba-2-780M · Commonsense reasoning average | accuracy: 31.0 -> 34.8 | Improved |
| replace log-linear fixed merging with DLA | Gated DeltaNet-1.3B · Commonsense reasoning average | accuracy: 32.6 -> 34.8 | Improved |
| replace log-linear with DLA | Mamba-2-780M · In-context retrieval average | accuracy: 6.3 -> 13.7 | Improved |
| replace log-linear with DLA | Gated DeltaNet-1.3B · In-context retrieval average | accuracy: 20.3 -> 28.8 | Improved |
| replace log-linear with DLA | Mamba-2-780M · RULER 4K S-NIAH-3 | accuracy: 10.6 -> 37.1 | Improved |
| add capacity-bounded memory modeling on top of dynamic state merging (ablation) | DLA(I) on Mamba-2 · RULER | RULER score: 44.0 -> 57.8 | Improved |
| add capacity-bounded memory modeling on top of dynamic state merging (ablation) | DLA(I) on Gated DeltaNet · RULER | RULER score: 58.4 -> 63.1 | Improved |
| add information-aware dynamic merging alone (DLA(I)) versus log-linear (ablation) | Mamba-2-780M · RULER | RULER score: 39.0 -> 44.0 | Improved |
| change cache capacity k from 30 to 20 or 40 (ablation) | DLA on Mamba-2 · RULER | RULER score: 57.8 -> 56.3 (k=20); 57.7 (k=40) | No clear effect |
| change merge boundary tau from 0.6 to 0.5 or 0.7 (ablation) | DLA on Mamba-2 · LAMBADA and RULER | accuracy / RULER score: LAMBADA 23.9 -> 20.1 (0.5), 22.5 (0.7); RULER 57.8 -> 57.3, 55.9 | Mixed |
| use DLA instead of log-linear for efficiency | Mamba-2-780M · Prefill throughput and memory | throughput and memory | Improved |
Closest papers by meaning
- Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.89)
- MVA: Linear Attention with High-order Query-Keys Integration and Multi-level Vocabulary Decomposition (ICML 2025 · similarity 0.88)
- Training-Free Long-Context Scaling of Large Language Models (ICML 2024 · similarity 0.88)
- LinearARD: Linear-Memory Attention Distillation for RoPE Restoration (NeurIPS 2026 · similarity 0.87)
- Quantifying Memory Utilization with Effective State-Size (ICML 2025 · similarity 0.87)
- Raven: High-Recall Sequence Modeling via Sparse Memory Routing (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
