Research map / Federated learning and privacy / Personalized and heterogeneous FL
Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DDOME, a decentralized federated MoE where a frozen pretrained common expert feeds a jointly trained gating network that routes clients to expert subsets, with anchor clients stabilizing specialization; theory shows disjoint expert/gate training fails.
Problem
Achieving expert specialization in MoEs under single-domain settings without explicit task boundaries, and personalizing to unseen clients in federated learning with heterogeneous data.
Approach
Jointly train a gating network (fed by embeddings from a frozen pretrained common expert) and a pool of experts across heterogeneous FL clients, using anchor clients (one per expert, sampled more often) to stabilize early specialization; clients receive only top-K experts, and test clients get per-sample routing to the highest-scored expert.
Key results
- On CIFAR10 DDOME reaches 91.8% (95.7% with the 93% common expert) vs. best baseline FedProx 72.7%/71.4%.
- On CIFAR100 DDOME reaches 75.7% (78.6% with the 73% common expert) vs. ~72.9% for FedAvg, a gain of about 2.8 to 4.6 points over the best baseline.
- On Yahoo! Answers DDOME reaches 64.55% (74.90% with the 69% common expert) vs. 62.42% (69.36%) for Average Ensembles.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DDOME instead of the strongest listed baseline | Federated zero-shot personalization · CIFAR10, CIFAR100, Yahoo! Answers, and EMNIST | accuracy: 91.8% vs 72.7%; 95.7% vs 71.4%; 75.7% vs 72.9%; 78.6% vs 74.0%; 64.55% vs 62.42%; 74.90% vs 69.36%; 79.9% vs 74.5%; 80.5 | Helped |
| Use DDOME instead of FedProx or FedAvg | Federated learning with Dirichlet label imbalance · CIFAR10 and CIFAR100, α=0.1 | global test accuracy: 80.8% vs 52.8%; 77.8% vs 73.6% | Helped |
| Initialize experts from the common expert instead of from scratch (ablation) | DDOME · CIFAR100 after 2000 rounds | zero-shot accuracy: 83.27% | Helped |
| Use a sufficiently capable common expert for gating and feature extraction (ablation) | DDOME · CIFAR100 | accuracy: 66%; 67%; 73% | Helped |
| Use 30% or more anchor clients per round rather than random clients only (ablation) | DDOME · CIFAR10 and CIFAR100 | global testing accuracy: 30% | Helped |
| Remove anchor clients from sampling (ablation) | DDOME · CIFAR10 and CIFAR100 | global testing accuracy | Hurt |
| Set the number of communicated experts equal to the total experts, M=K (ablation) | DDOME · CIFAR10 Dirichlet α=0.1 | best global test accuracy: 60.16%; 75.77% | Mixed |
| Train DDOME while dynamically increasing the client pool (ablation) | DDOME · Incremental CIFAR100 | accuracy: approximately 4% | Helped |
| Switch the active client pool every 600 iterations (ablation) | DDOME · Incremental CIFAR100 | accuracy: approximately 2% | Mixed |
| Train gating and experts jointly rather than sequentially | MoE with top-1 routing and linear experts · Gaussian-input theoretical setting | expected loss: Ω(1) | Helped |
Surprising findings
- FedAvg fails to improve over the pretrained common expert in several settings, and Scaffold became unstable and produced useless models.
- With a common expert below about 66-67% accuracy on CIFAR100, the gating collapses to selecting a single expert and fails to beat the baseline.
Limitations
- Depends on the availability of an effective pretrained common expert; the method fails below about 66% common-expert accuracy on CIFAR100.
- Higher communication overhead than single-model FL methods, and limited scalability testing to a moderate number of clients.
- Anchor-client mechanism requires some a priori knowledge of client diversity; validation limited to classification tasks with homogeneous expert architectures.
Experimental setup
Datasets: CIFAR10 CIFAR100 EMNIST Yahoo! Answers
Benchmarks: CIFAR10 CIFAR100 EMNIST Yahoo! Answers
Metrics: accuracy zero-shot personalization accuracy on unseen test clients
Baselines: FedMix FedAvg FedProx Scaffold Average Ensembles FedSR FedADG
Builds on: FedAvg FedProx FedMix Scaffold
Closest papers by meaning
- Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts (NeurIPS 2024 · similarity 0.88)
- Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations (NeurIPS 2025 · similarity 0.88)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.87)
- Autonomy-of-Experts Models (ICML 2025 · similarity 0.87)
- Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (NeurIPS 2024 · similarity 0.87)
- DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.