Research map / Architectures and efficiency / Mixture of experts
Tight Clusters Make Specialized Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Adaptive Clustering (AC) router for sparse MoE, which scales token features by cluster-wise inverse-dispersion weights derived from a feature-weighted clustering objective, improving convergence, robustness and downstream performance without learnable parameters.
Problem
Standard MoE routers rely on dot-products between tokens and expert embeddings, which can fail to discover latent clusters in high-dimensional space, leading to slow convergence, poor robustness to contamination, and degraded representations.
Approach
Derive optimal cluster-wise feature weights from a KL-regularized feature-weighted clustering objective (weights inversely proportional to per-cluster feature dispersion), and compute routing scores in a cluster-adaptive transformed space using the previous layer's expert assignments.
Key results
- On WikiText-103 pretraining, GLaM-ACMoE reaches 36.26 test PPL vs 38.27 for GLaM (about 4.8% relative improvement at medium size), and Switch-ACMoE reaches 34.42 vs 35.48.
- On ImageNet under PGD attack, Swin-ACMoE-Top 2 reaches 43.74 top-1 vs 40.85 for Swin-Top 2 (about 7% gain).
- Efficiency overhead is within about 1% in speed and memory, with no added learnable parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard routing with the Adaptive Clustering router | Switch · WikiText-103 pretrain | PPL: 35.48 -> 34.42 | Helped |
| Replace standard routing with the Adaptive Clustering router | GLaM · WikiText-103 pretrain | PPL: 38.27 -> 36.26 | Helped |
| Replace standard routing with the Adaptive Clustering router | Switch · EnWik-8 pretrain | BPC: 1.153 -> 1.137 | Helped |
| Pretrain with the Adaptive Clustering router instead of standard routing | Switch · SST2, SST5 and Banking-77 finetuning after EnWik-8 pretraining | top-1 accuracy: SST2: 63.27 -> 64.45; SST5: 32.21 -> 33.79; B77: 53.48 -> 54.26 | Helped |
| Pretrain with the Adaptive Clustering router instead of standard routing | Switch · SST2, SST5 and Banking-77 finetuning after WikiText-103 pretraining | top-1 accuracy: SST2: 76.27 -> 77.32; SST5: 39.13 -> 40.04; B77: 83.82 -> 86.01 | Helped |
| Pretrain with the Adaptive Clustering router instead of standard routing | GLaM · SST2, SST5 and Banking-77 finetuning after WikiText-103 pretraining | top-1 accuracy: SST2: 69.97 -> 71.90; SST5: 33.69 -> 34.24; B77: 80.89 -> 82.33 | Helped |
| Replace standard routing with the Adaptive Clustering router | Switch · WikiText-103 contaminated test | PPL: 48.12 -> 47.61 | Helped |
| Replace standard routing with the Adaptive Clustering router | GLaM · WikiText-103 contaminated test | PPL: 50.84 -> 47.91 | Helped |
| Use Adaptive Clustering routing with top-2 rather than standard Swin routing | Swin Transformer · ImageNet clean | top-1 accuracy: 76.10 -> 76.31 | Helped |
| Use Adaptive Clustering routing with top-2 rather than standard Swin routing | Swin Transformer · ImageNet PGD, FGSM, and SPSA | top-1 accuracy: PGD: 40.85 -> 43.74; FGSM: 54.70 -> 55.78; SPSA: 60.57 -> 63.47 | Helped |
| Use Adaptive Clustering routing with top-1 rather than standard Swin routing | Swin Transformer · ImageNet PGD and FGSM | top-5 accuracy: PGD: 74.59 -> 73.46; FGSM: 83.86 -> 82.80 | Hurt |
| Use Adaptive Clustering routing with top-1 rather than standard Swin routing | Swin Transformer · ImageNet SPSA | top-5 accuracy: 82.63 -> 82.47 | Hurt |
Surprising findings
- The AC router is not applied to the first MoE layer, since it needs previous-layer expert assignments.
Limitations
- Requires previous-layer expert assignments as cluster estimates, so the embedding size must stay the same between adjacent MoE layers.
- Not implementable in the first MoE layer.
- Robustness and convergence results rely on Gaussian mixture assumptions and some approximations (e.g., U approximately identity).
Experimental setup
Datasets: WikiText-103 EnWik-8 SST2 SST5 Banking-77 ImageNet-1K ImageNet-A ImageNet-O ImageNet-R
Benchmarks: WikiText-103 EnWik-8 SST2 SST5 Banking-77 ImageNet-1K ImageNet-A ImageNet-O ImageNet-R
Metrics: perplexity bits-per-character top-1 accuracy top-5 accuracy AUPR load balance standard deviation throughput max memory
Baselines: Switch Transformer GLaM Swin Transformer SMoE router XMoE StableMoE
Builds on: Switch Transformer GLaM Swin Transformer XMoE feature-weighted clustering
Closest papers by meaning
- Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts (NeurIPS 2026 · similarity 0.90)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.89)
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts (NeurIPS 2025 · similarity 0.89)
- Layerwise Recurrent Router for Mixture-of-Experts (ICLR 2025 · similarity 0.89)
- Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning (ICML 2025 · similarity 0.88)
- Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.