Research map / Architectures and efficiency / Mixture of experts
Statistical Advantages of Perturbing Cosine Router in Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis of least-squares estimation in cosine-router MoE showing slow (log-rate) expert estimation, and proposes a perturbed cosine router (noise added to L2 norms) that restores polynomial rates, validated with synthetic, language modeling, and domain generalization experiments.
Problem
Cosine routers in Mixture of Experts are empirically effective, but their statistical convergence properties for expert and parameter estimation are not understood; the normalization induces parameter interactions that may slow estimation.
Approach
Show that the cosine router's normalization creates a PDE-based parameter interaction causing non-polynomial (log-rate) estimation; adding noise to the L2 norms (perturbed cosine router) removes this interaction and yields polynomial rates under strong identifiability of experts.
Key results
- Perturbed cosine router achieves empirical rate O(n^-0.5) in exact-specified synthetic setting versus O(n^-0.11) for vanilla cosine router.
- On Enwik8 (medium) BPC improves from 1.161 (cosine) to 1.147 (perturbed); Wikitext-103 medium PPL 38.018 to 37.859.
- DomainNet-style DomainBed average accuracy rises from 66.70 (cosine) to 68.37 (perturbed), vs 65.40 for linear router.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add noise to the norms in the cosine router | Cosine-router MoE · Enwik8 Small | BPC: 1.213 -> 1.197 | Helped |
| Add noise to the norms in the cosine router | Cosine-router MoE · Enwik8 Medium | BPC: 1.161 -> 1.147 | Helped |
| Add noise to the norms in the cosine router | Cosine-router MoE · Text8 Small | BPC: 1.310 -> 1.303 | Helped |
| Add noise to the norms in the cosine router | Cosine-router MoE · Text8 Medium | BPC: 1.271 -> 1.251 | Helped |
| Add noise to the norms in the cosine router | Cosine-router MoE · Wikitext-103 Small | PPL: 90.070 -> 89.910 | Helped |
| Add noise to the norms in the cosine router | Cosine-router MoE · Wikitext-103 Medium | PPL: 38.018 -> 37.859 | Helped |
| Add noise to the norms in the cosine router | MoE with ViT-S/16 · PACS | accuracy: 87.22 -> 89.36 | Helped |
| Add noise to the norms in the cosine router | MoE with ViT-S/16 · VLCS | accuracy: 78.99 -> 80.01 | Helped |
| Add noise to the norms in the cosine router | MoE with ViT-S/16 · OfficeHome | accuracy: 73.27 -> 74.09 | Helped |
| Add noise to the norms in the cosine router | MoE with ViT-S/16 · TerraIncognita | accuracy: 45.55 -> 49.87 | Helped |
| Add noise to the norms in the cosine router | MoE with ViT-S/16 · DomainNet | accuracy: 48.45 -> 48.51 | Helped |
| Add noise to the norms in the cosine router | MoE with ViT-S/16 · Average across DomainBed datasets | accuracy: 66.70 -> 68.37 | Helped |
Surprising findings
- Vanilla cosine router yields non-polynomial (log-rate) expert estimation, even slower than linear router for polynomial experts.
- Improvements on Wikitext-103 are only slight.
Limitations
- Theory assumes well-specified data generation; misspecified setting left for future work.
- Rates are pointwise rather than uniform over parameters.
- Code to be released only upon acceptance (not yet available).
Experimental setup
Datasets: Enwik8 Text8 Wikitext-103 PACS VLCS OfficeHome TerraIncognita DomainNet
Benchmarks: Enwik8 Text8 Wikitext-103 PACS VLCS OfficeHome TerraIncognita DomainNet
Metrics: Voronoi loss convergence rate BPC perplexity out-of-distribution accuracy
Baselines: linear router cosine router Switch Transformer
Builds on: Switch Transformer cosine router MoE Voronoi loss for mixture models Gaussian MoE expert estimation
Closest papers by meaning
- Layerwise Recurrent Router for Mixture-of-Experts (ICLR 2025 · similarity 0.88)
- Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts (NeurIPS 2026 · similarity 0.87)
- ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing (ICLR 2025 · similarity 0.86)
- Tight Clusters Make Specialized Experts (ICLR 2025 · similarity 0.86)
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts (NeurIPS 2025 · similarity 0.85)
- Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.