Research map / Architectures and efficiency / Attention and transformers

A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse and structured attention · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows each entry of gated-attention and multi-head self-attention matrices is a hierarchical mixture of experts, and uses this to prove that gated attention (non-linearity after value projection or SDPA output) needs polynomial sample complexity for expert estimation, versus exponential for vanilla MHA.

Problem

Gated attention has shown empirical gains (reduced attention sink, higher expressiveness of low-rank value-output mapping), but lacks theoretical justification; the question is why and where gating helps in terms of statistical sample efficiency.

Approach

Recasts MHA and gated attention as hierarchical mixture-of-experts models and treats learning as expert estimation; shows the PDE-type parameter interaction from linear experts causes exponential sample complexity in MHA, and that a non-linear activation after the value projection or SDPA output (type-1/type-2 strong identifiability) removes it, giving polynomial O(eps^-4) complexity.

Key results

• Empirical Voronoi loss decays roughly as O(n^-0.49) to O(n^-0.53) for gated attention Settings I and II versus O(n^-0.12) to O(n^-0.13) for standard multi-head self-attention.
• Theory: MHA expert estimation requires O(exp(eps^-1/tau)) samples, while gated attention Settings I and II require O(eps^-4).

Surprising findings

• Standard MHA parameter estimation is slower than any polynomial rate, due to a PDE-type interaction induced by linear experts.

Limitations

• Analysis assumes well-specified regression where data follow an HMoE model.
• Compares statistical expert estimation rather than universal approximation or downstream performance.
• Experiments are small synthetic settings (d=2, H=2, N=2, K=2) rather than real Transformers.

Experimental setup

Datasets: synthetic regression data (uniform inputs on [-1,1]^2, Gaussian noise)

Benchmarks: synthetic HMoE regression

Metrics: empirical Voronoi loss convergence rate exponent

Baselines: multi-head self-attention value gated attention SDPA gated attention

Builds on: gated attention hierarchical mixture of experts Voronoi loss mixture of experts expert specialization

Closest papers by meaning

• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (NeurIPS 2025 · similarity 0.88)
• Bayes optimal learning of attention-indexed models (NeurIPS 2025 · similarity 0.87)
• MoH: Multi-Head Attention as Mixture-of-Head Attention (ICML 2025 · similarity 0.86)
• Mixture-of-Top-$k$ Attention: Efficient Attention as Scalable Fast Weights (NeurIPS 2026 · similarity 0.86)
• Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems (NeurIPS 2025 · similarity 0.86)
• Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers (NeurIPS 2024 · similarity 0.85)