Research map / Architectures and efficiency / Mixture of experts
Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
Zizhuo Fu, Wenxuan Zeng, Runsheng Wang, Meng Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add auxiliary head load balancing loss (ablation) | Vanilla attention 0.6B · from scratch 11-benchmark average | accuracy: 39.54 -> 40.82 | Improved |
| add auxiliary head load balancing loss (ablation) | Gated attention 2B · from scratch 11-benchmark average | accuracy: 46.30 -> 47.70 | Improved |
| add auxiliary head load balancing loss (ablation) | Sink attention 2B · from scratch 11-benchmark average | accuracy: 46.35 -> 46.50 | Improved |
| add auxiliary head load balancing loss (ablation) | Vanilla attention 2B · FineWeb-Edu validation | BPB: 0.7243 -> 0.7198 | Improved |
| use Gated attention instead of Vanilla attention | base loss, 2B · from scratch 11-benchmark average | accuracy: 45.46 -> 46.30 | Improved |
| use Sink attention instead of Vanilla attention | base loss, 0.6B · from scratch 11-benchmark average | accuracy: 39.54 -> 40.31 | Improved |
| add auxiliary load balancing loss in fine-tuning (ablation) | Qwen3-8B · reasoning average | accuracy: 74.31 -> 76.01 | Improved |
| add auxiliary load balancing loss in fine-tuning (ablation) | LLaMA3.1-8B · reasoning average ProcessBench | accuracy: 23.63 -> 31.14 | Improved |
| add auxiliary load balancing loss in fine-tuning (ablation) | Qwen3-8B · LongBench average | accuracy: 50.67 -> 55.38 | Improved |
| add auxiliary load balancing loss in fine-tuning (ablation) | Qwen3-4B · LongBench HotpotQA | accuracy: 60.81 -> 58.61 | Hurt |
| add auxiliary load balancing loss in fine-tuning (ablation) | LLaMA3.1-8B · LongBench TriviaQA | accuracy: 46.34 -> 41.82 | Hurt |
| zero first-token value in all heads (ablation) | Qwen3-8B · 9-benchmark average | accuracy: 70.36 -> 64.10 | Hurt |
Closest papers by meaning
- When Attention Sink Emerges in Language Models: An Empirical View (ICLR 2025 · similarity 0.92)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.91)
- On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs (NeurIPS 2026 · similarity 0.90)
- Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings (NeurIPS 2025 · similarity 0.88)
- MoH: Multi-Head Attention as Mixture-of-Head Attention (ICML 2025 · similarity 0.88)
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
