Research map / Architectures and efficiency / Attention and transformers
Householder-Diagonalized Linear Attention (HDLA): Utilizing Enhanced Decay Mechanism for Efficient Sequence Modeling
Jiefu Zhang, Zhen Qin, Jiabo Tong, Shijie Mei, JiaKui Hu, Yuqi Pan, Anjie Hu, Man Yao, Bo XU, Guoqi Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| HDLA decay vs Gated DeltaNet | linear attention · MAD (2048) | average score: 68.17 -> 72.97 | Improved |
| Expand hidden state from 2048 to 4096 (ablation) | HDLA · MAD | average score: 72.97 -> 74.17 | Improved |
| Use HDLA instead of Gated DeltaNet at 2.8B | linear attention LM · FineWeb-Edu avg perplexity (2.8B) | perplexity: 19.60 -> 18.58 | Improved |
| Use HDLA instead of Llama at 2.8B | softmax baseline · FineWeb-Edu avg perplexity (2.8B) | perplexity: 20.71 -> 18.58 | Improved |
| Use HDLA instead of Gated DeltaNet on S-NIAH-3 | 1.45B model · S-NIAH-3 length 2048 | accuracy: 7.0% -> 65.2% | Improved |
| Use HDLA instead of Gated DeltaNet on S-NIAH-3 | 1.45B model · S-NIAH-3 length 1024 | accuracy: 50.6% -> 82.0% | Improved |
| Expand hidden state (nh 12 to 4) (ablation) | HDLA 0.4B · AVG-RET | accuracy: 10.54 -> 13.91 | Improved |
| Expand hidden state (nh 12 to 3) (ablation) | HDLA 0.17B · AVG-RET | accuracy: 7.20 -> 9.62 | Improved |
| Restrict beta range from [0,2] to [0,1] (ablation) | HDLA 0.4B · AVG-RET | accuracy: 11.71 -> 12.51 | No clear effect |
| Raise learning rate from 3e-4 to 6e-4 (ablation) | HDLA 0.4B · AVG-RET | accuracy: 11.7 -> 13.3 | Improved |
| Use SiLU vs 1+elu on keys/values (ablation) | HDLA 0.4B · AVG-RET | accuracy: 11.71 -> 12.93 | No clear effect |
| Use HDLA instead of Gated DeltaNet | linear attention · Training throughput | tokens per second: 19277.18 -> 7340.01 | Hurt |
Closest papers by meaning
- MVA: Linear Attention with High-order Query-Keys Integration and Multi-level Vocabulary Decomposition (ICML 2025 · similarity 0.90)
- Efficient Low Rank Attention for Long-Context Inference in Large Language Models (NeurIPS 2025 · similarity 0.88)
- SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024 · similarity 0.88)
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.88)
- Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences (ICML 2024 · similarity 0.87)
- HyperAttention: Long-context Attention in Near-Linear Time (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
