Research map / Architectures and efficiency / Attention and transformers
MHLA: Restoring Expressivity of Linear Attention via Token-Level Multi-Head
Kewei Zhang, Ye Huang, Yufan Deng, Jincheng YU, Junsong Chen, Huan Ling, Enze Xie, Daquan Zhou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace linear attention with MHLA | DeiT-T · ImageNet-1K | top-1 accuracy: 69.8 -> 75.8 | Improved |
| replace self attention with MHLA | DeiT-T · ImageNet-1K | top-1 accuracy: 72.2 -> 75.8 | Improved |
| replace self attention with MHLA | DeiT-S · ImageNet-1K | top-1 accuracy: 79.8 -> 81.0 | Improved |
| add MHLA at 384x384 resolution | DeiT-T · ImageNet-1K 384 | top-1 accuracy: 74.4 -> 77.5 | Improved |
| add MHLA at 512x512 resolution | DeiT-T · ImageNet-1K 512 | top-1 accuracy: 75.3 -> 78.3 | Improved |
| replace linear attention with MHLA | DiT-S/2 · ImageNet C2I 512px | FID: 125.33 -> 78.63 | Improved |
| replace self attention with MHLA | DiT-S/2 · ImageNet C2I 512px | FID: 84.54 -> 78.63 | Improved |
| replace self attention with MHLA with CFG | DiT-XL/2 fine-tuned · ImageNet C2I | FID: 2.27 -> 2.54 | Hurt |
| add CPE to MHLA (ablation) | MHLA · DiT-S/2 256px | FID: 76.4 -> 64.0 | Improved |
| add output gating to MHLA (ablation) | MHLA · DiT-S/2 256px | FID: 76.4 -> 68.5 | Improved |
| add CPE and gating together to MHLA (ablation) | MHLA · DiT-S/2 256px | FID: 76.4 -> 59.8 | Improved |
| add CPE to MHLA (ablation) | MHLA · DiT-XL/2 256px | FID: 20.32 -> 22.79 | Hurt |
Closest papers by meaning
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.88)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.88)
- MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map (NeurIPS 2024 · similarity 0.87)
- STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs (NeurIPS 2026 · similarity 0.87)
- Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few (NeurIPS 2025 · similarity 0.86)
- Gated Linear Attention Transformers with Hardware-Efficient Training (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
