Research map / Architectures and efficiency / Mixture of experts
On Linear Mode Connectivity of Mixture-of-Experts Architectures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes the permutation/translation symmetries of dense and sparse Mixture-of-Experts gating and proposes a weight-matching algorithm that aligns independently trained MoEs, empirically showing linear mode connectivity in ViT and GPT-2 MoE replacements.
Problem
Whether linear mode connectivity (LMC) up to permutation holds for Mixture-of-Experts architectures, whose symmetries (expert permutations and gating translations) differ from standard feedforward networks, and how to align independently trained MoEs to reveal it.
Approach
Proves that the functional symmetries of dense and sparse (Top-k) MoE gating are fully captured by a group action permuting experts and translating gating parameters, then uses a two-stage LAP-based weight matching (expert ordering via gate or expert weights, then per-expert neuron alignment) to align MoEs so that linear interpolation shows low loss barriers.
Key results
- Two-stage weight matching yields near-optimal expert ordering: on 12-layer ViT-MoE with 4 experts on CIFAR-10, the expert-weight method ranks about 2.5 of 24 permutations at layer 1 with scaled loss barrier about 2.1.
- LMC is observed across dense, SMoE, and DeepSeekMoE variants on ViT (MNIST, CIFAR, ImageNet) and GPT-2 (WikiText103, One Billion Word), with the loss barrier reduced relative to naive interpolation.
- Total Weight Matching gives consistently lower loss barriers than Skipping-Expert-Order Matching on One Billion Word across MoE variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Align expert order and internal weights using Expert Weight Matching | ViT-MoE, 4 experts · CIFAR-10, layer 1 | L̂: 2.12 ± 0.42 | Helped |
| Align expert order and internal weights using Gate Weight Matching | ViT-MoE, 4 experts · CIFAR-10, layer 1 | L̂: 3.42 ± 0.55 | Helped |
| Replace the first Transformer layer with an MoE rather than the second (ablation) | ViT-MoE on CIFAR-10, 2 layers · CIFAR-10 | Loss barrier ratio: 8.54 ± 1.53 vs 9.33 ± 2.04 | Helped |
| Replace the last Transformer layer with an MoE rather than the first (ablation) | ViT-MoE on CIFAR-10, 4 layers · CIFAR-10 | Loss barrier ratio: 5.12 ± 0.67 vs 8.29 ± 1.63 | Helped |
| Replace the first Transformer layer with an MoE rather than the fourth (ablation) | ViT-MoE on CIFAR-10, 6 layers · CIFAR-10 | Loss barrier ratio: 4.16 ± 1.47 vs 10.83 ± 5.61 | Helped |
| Use Total Weight Matching instead of Skipping-Expert-Order Matching | MoE interpolation on One Billion Word · One Billion Word | loss barrier | Helped |
Surprising findings
- LMC is less pronounced when the MoE replaces the last Transformer layer than the first, suggesting early layers shape the solution basin more.
Limitations
- No theoretical bound on the loss barrier is provided.
- The k=1 sparse gating case is excluded due to additional invariances and deferred to future work.
- Expert internal ReLU symmetries are not fully characterized; experts are treated as black boxes in the functional equivalence results.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100 ImageNet-21k ImageNet-1k WikiText103 One Billion Word
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k WikiText103 One Billion Word
Metrics: loss barrier loss AUC accuracy barrier accuracy AUC expert-order rank scaled loss barrier
Baselines: Skipping-Expert-Order Matching naive interpolation
Builds on: Git Re-Basin weight matching linear mode connectivity DeepSeekMoE Sinkhorn re-basin
Closest papers by meaning
- Generalized Linear Mode Connectivity for Transformers (NeurIPS 2025 · similarity 0.89)
- On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks (NeurIPS 2025 · similarity 0.88)
- MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts (AAAI 2024 · similarity 0.88)
- Linear Mode Connectivity in Differentiable Tree Ensembles (ICLR 2025 · similarity 0.87)
- Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning (ICML 2025 · similarity 0.87)
- Understanding Mode Connectivity via Parameter Space Symmetry (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.