Research map / Architectures and efficiency / Attention and transformers
Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically compares 30 gating variants in softmax attention on 15A2B MoE and 1.7B dense models trained on up to 3.5T tokens, finding a head-specific sigmoid gate after SDPA improves performance, stability, and long-context extrapolation by adding non-linearity and input-dependent sparsity that removes attention sinks.
Problem
Gating is widely used in neural networks but its specific effects in softmax attention are poorly understood, and prior gated-attention comparisons are confounded by other architectural factors.
Approach
Apply a head-specific, query-dependent sigmoid gate to the scaled dot-product attention output (G1). The gate adds non-linearity to the low-rank W_V W_O mapping and induces input-dependent sparsity that suppresses attention sinks and massive activations, improving stability and context-length extension.
Key results
- SDPA elementwise sigmoid gate lowers average PPL from 6.026 to 5.761 on the 15A2B MoE (400B tokens) and raises MMLU from 58.79 to 60.82.
- On the 1.7B dense model trained on 3.5T tokens, SDPA gating improves PPL from 6.180 to 6.130 and nearly eliminates loss spikes, enabling a larger max LR.
- With YaRN extension to 128k, the gated model reaches 58.82 on RULER versus 31.65 for baseline, while the first-token attention share drops from 46.7% to 4.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add head-specific elementwise sigmoid gating after SDPA | 15A2B MoE · 400B tokens | Avg PPL: 6.026 -> 5.761 | Helped |
| Add elementwise sigmoid gating after SDPA | 1.7B dense · 400B tokens | Avg PPL: 7.499 -> 7.404 | Helped |
| Add elementwise sigmoid gating after SDPA | 1.7B dense · 3.5T tokens | Avg PPL: 6.180 -> 6.130 | Helped |
| Add SDPA output gating | 1.7B dense, 48 layers · 1T tokens, max LR 8.0 × 10^-3 | training convergence | Helped |
| Move the gate from SDPA output to value projection (ablation) | 15A2B MoE · 400B tokens | Avg PPL: 5.761 -> 5.820 | Hurt |
| Move the gate to key projection (ablation) | 15A2B MoE · 400B tokens | Avg PPL: 5.761 -> 6.016 | Hurt |
| Move the gate to query projection (ablation) | 15A2B MoE · 400B tokens | Avg PPL: 5.761 -> 5.981 | Hurt |
| Move gating after the dense output layer (ablation) | 15A2B MoE · 400B tokens | Avg PPL: 5.761 -> 6.017 | Hurt |
| Use headwise rather than elementwise gating (ablation) | 15A2B MoE · 400B tokens | Avg PPL: 5.761 -> 5.792 | Hurt |
| Share gate scores across attention heads (ablation) | 15A2B MoE · 400B tokens | Avg PPL: 5.761 -> 5.801 | Hurt |
| Use additive SiLU gating instead of multiplicative sigmoid gating (ablation) | 15A2B MoE · 400B tokens | Avg PPL: 5.761 -> 5.821 | Hurt |
| Replace sigmoid with SiLU for multiplicative gating (ablation) | 15A2B MoE · 400B tokens | Avg PPL: 5.761 -> 5.822 | Hurt |
Surprising findings
- Gating at the value projection removes massive activations but attention sinks persist, indicating massive activations are not a prerequisite for attention sinks.
- Substantial gains persist for Switch Heads even when reduced to a single expert, suggesting the gating itself, not routing, drives the improvement.
Limitations
- Non-linearity effects on attention dynamics and overall training remain under-explored.
- No rigorous theoretical explanation of how attention sinks affect length generalization.
- Mechanistic explanation of training stability gains is partly hypothesized rather than fully proven.
Experimental setup
Datasets: 3.5T high-quality token pretraining corpus (multilingual, math, general knowledge)
Benchmarks: Hellaswag MMLU GSM8k HumanEval C-eval CMMLU RULER
Metrics: perplexity accuracy RULER score
Baselines: Baseline MoE (q32, kv4) Switch Heads sandwich norm Forgetting Transformer Quantizable Transformers gated attention
Builds on: softmax attention Switch Heads Native Sparse Attention YaRN sandwich normalization
Closest papers by meaning
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.89)
- When Attention Sink Emerges in Language Models: An Empirical View (ICLR 2025 · similarity 0.89)
- MVA: Linear Attention with High-order Query-Keys Integration and Multi-level Vocabulary Decomposition (ICML 2025 · similarity 0.89)
- On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models (NeurIPS 2026 · similarity 0.88)
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.88)
- Liger: Linearizing Large Language Models to Gated Recurrent Structures (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.