Research map / Architectures and efficiency / Mixture of experts
Autonomy-of-Experts Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Autonomy-of-Experts (AoE), an MoE paradigm that removes routers so experts self-select by the L2 norm of a low-rank cached activation, and pre-trains 732M and 4B MoE-style LLMs to show gains over standard MoE.
Problem
Standard MoE uses a separate router to assign tokens to experts, so routing is a prediction without direct feedback on expert ability, leading to suboptimal expert selection and inefficient learning.
Approach
Remove routers; each expert pre-computes a low-rank activation (via factorized W_g = W_down W_up) for every token, and experts are ranked by the L2 norm of their activations, with only top-K experts completing the forward pass. This lets experts self-evaluate and compare with peers.
Key results
- On 732M-parameter models (247M active, 100B tokens), AoE with d_low=256 and auxiliary loss reaches 44.46 average accuracy vs. 43.68 for the best traditional MoE baseline with auxiliary loss.
- On 4B-parameter models (1.18B active), AoE achieves 49.80 average accuracy vs. 48.06 for traditional MoE.
- AoE reaches up to 97% of traditional MoE training throughput (46.07-49.79 K/s vs. 51.42 K/s) at higher memory cost (55.9-59.4 GB vs. 50.6 GB).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace router-based selection with autonomous activation-norm selection | 732M language model · Eight-task average accuracy | accuracy: 43.68 -> 44.39 | Helped |
| replace router-based selection with autonomous activation-norm selection | 4B language model · Eight-task average accuracy | accuracy: 48.06 -> 49.80 | Helped |
| add auxiliary loss (ablation) | AoE · Eight-task average accuracy across d_low settings | accuracy: 43.81 -> 44.12; 44.39 -> 44.04; 44.12 -> 44.46; 43.77 -> 44.08 | Mixed |
| vary d_low from 64 to 128, 256, and 512 (ablation) | AoE · Eight-task average accuracy | accuracy: 43.81; 44.39; 44.12; 43.77 | Mixed |
| factorize Wg (ablation) | Traditional MoE with auxiliary loss · Eight-task average accuracy | accuracy: 43.68 -> 43.70 | No clear effect |
| use a larger router with selection parameter count matched to AoE (ablation) | Traditional MoE with auxiliary loss · Eight-task average accuracy | accuracy: 43.68 -> 43.71 | Helped |
| use Top-P selection instead of Top-K selection (ablation) | AoE versus traditional MoE · Eight-task average accuracy, Top-P | accuracy: 43.87 -> 44.29 | Helped |
| use expert-choice selection instead of Top-K selection (ablation) | AoE versus traditional MoE · Eight-task average accuracy, expert-choice | accuracy: 43.89 -> 44.17 | Helped |
| select experts using L1 or L-infinity activation norms instead of L2 norms (ablation) | Pre-trained MoE-LLMs · MMLU and ARC-C, 5-shot | accuracy | Hurt |
| increase d_low (ablation) | AoE · Training throughput | K tokens/s: 49.79 -> 49.42 -> 47.98 -> 46.07 | Hurt |
Surprising findings
- Removing routers from pre-trained Mixtral 8x7B and selecting experts by activation norm without any parameter updates preserved up to about 95% of MMLU/ARC-C accuracy at some pause nodes.
- AoE without any auxiliary loss already outperforms traditional MoE models, while the auxiliary loss gives further gains in some configurations.
Limitations
- Efficiency falls as the number of experts increases and sparsity grows, and memory usage is higher than traditional MoE.
- Gains on the small-model ablations are modest (e.g., about 0.1 to 0.8 average accuracy points), and improvements are within small margins for several tasks.
- Scaling experiments are limited to 4B total parameters; the authors say efficiency under more experts is still being optimized.
Experimental setup
Datasets: RedPajama
Benchmarks: ARC-easy PIQA SIQA Winogrande HellaSwag MNLI MRPC QNLI QQP SST-2 MMLU ARC-Challenge
Metrics: accuracy average accuracy throughput (tokens/s per GPU) memory usage (GB) training NLL loss load entropy confidence entropy
Baselines: Traditional MoE with top-K router Traditional MoE with auxiliary loss Top-P token-choice MoE Expert-choice MoE Mixtral 8x7B Phi-3.5-MoE-instruct
Builds on: Mixture-of-Experts (sparse top-K routing) Switch Transformer load-balancing loss Expert-Choice routing Llama FFN (SwiGLU) expert design FFN as key-value memories
Closest papers by meaning
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.91)
- MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts (AAAI 2024 · similarity 0.90)
- Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast (NeurIPS 2024 · similarity 0.89)
- Multi-Head Mixture-of-Experts (NeurIPS 2024 · similarity 0.89)
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.88)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.