Research map / Architectures and efficiency / Mixture of experts
Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Multilinear Mixture of Experts (μMoE), a differentiable MoE layer whose expert weight tensor is kept in CP or Tensor-Ring factorized form so that thousands of experts can be computed densely, and shows increasing expert count yields more class-specialized experts in vision fine-tuning and parameter-matched pre-training of GPT-2 and MLP-Mixer.
Problem
Sparse MoEs use discrete top-K routing that is non-differentiable and unstable, while dense MoEs with many experts are parameter- and compute-prohibitive, limiting fine-grained expert specialization for interpretability and editing.
Approach
Parameterize the full expert weight tensor (N x I x O) with CP or Tensor-Ring factorization and compute the dense, softmax-like (entmax) weighted mixture of all N experts directly in factorized form, never materializing the tensor, so the layer is differentiable and scales to tens of thousands of experts at parameter parity with a dense MLP.
Key results
- On CLIP ViT-B-32 fine-tuned on ImageNet, mean expert class-level polysemanticity drops steadily as experts increase from 32 to 1024.
- On CelebA, expert-conditional re-writing (expert thresh.) raises target subpopulation accuracy for old females from 0.516 (linear) to 0.866 and for blond males from 0.346 to 0.847 without retraining.
- Parameter-matched μMoE pretraining matches MLP baselines: MLP-Mixer S-16 val acc 70.31 (MLP) vs 71.29 (CP μMoE); CLIP fine-tuning 77.99 (MLP) vs 78.71 (TR μMoE).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace MLP blocks with parameter-matched TRμMoE blocks | MLP-Mixer S-16 · ImageNET1k | validation accuracy: 70.31 -> 71.26 | Helped |
| replace MLP blocks with parameter-matched CPμMoE blocks | MLP-Mixer S-16 · ImageNET1k | validation accuracy: 70.31 -> 71.29 | Helped |
| replace MLP blocks with parameter-matched TRμMoE blocks | GPT-2 NanoGPT · OWT | validation loss: 2.876 -> 2.886 | Hurt |
| replace MLP blocks with parameter-matched CPμMoE blocks | GPT-2 NanoGPT · OWT | validation loss: 2.876 -> 2.893 | Hurt |
| replace the dense MLP with TRμMoE | CLIP ViT-B-32 · ImageNET1k | validation accuracy: 77.99 -> 78.71 | Helped |
| replace the dense MLP with CPμMoE | CLIP ViT-B-32 · ImageNET1k | validation accuracy: 77.99 -> 78.07 | Helped |
| increase the number of μMoE experts | CPμMoE fine-tuned on ImageNET1k · ImageNET1k | class-level expert polysemanticity | Helped |
| apply expert-conditional threshold rewriting | CPμMoE on CelebA Old females / Age · CelebA Old females / Age | target subpopulation accuracy: 0.555 -> 0.866 | Helped |
| apply expert-conditional threshold rewriting | CPμMoE on CelebA Old females / Age · CelebA Old females / Age | test set accuracy: 89.048 -> 84.650 | Hurt |
| apply expert-conditional threshold rewriting | CPμMoE on CelebA Blond males / Blond Hair · CelebA Blond males / Blond Hair | target subpopulation accuracy: 0.409 -> 0.847 | Helped |
| apply expert-conditional threshold rewriting | CPμMoE on CelebA Blond males / Blond Hair · CelebA Blond males / Blond Hair | test set accuracy: 95.893 -> 94.895 | Hurt |
Surprising findings
- Expert re-writing bias correction on CelebA outperformed standard fairness baselines in most settings without any re-training.
- Truncating up to half of the singular vectors of pre-trained MLP-Mixer linear layers leaves validation accuracy nearly unchanged, suggesting low-rank MLPs suffice.
Limitations
- Quantitative specialization evaluation covers only test-set class-level behavior, not out-of-distribution data.
- Fine-grained specialism in large language and vision models is shown only qualitatively due to lack of fine-grained labels.
- Comparable accuracy was demonstrated only for models around 100M parameters.
Experimental setup
Datasets: ImageNet1k CelebA OpenWebText
Benchmarks: ImageNet1k CelebA OpenWebText
Metrics: class-level polysemanticity top-1 accuracy validation loss equality of opportunity STD bias subpopulation max-min target subpopulation accuracy
Baselines: Dense MoE Sparse MoE Linear layer / MLP HighRankLinear oversampling adversarial debiasing blind thresholding
Builds on: Mixture of Experts entmax tensor-train decomposition robust fine-tuning (WiSE-FT style protocol) MLP-Mixer
Closest papers by meaning
- Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks (CVPR 2025 · similarity 0.90)
- DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models (CVPR 2025 · similarity 0.90)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.89)
- On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks (NeurIPS 2025 · similarity 0.89)
- MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts (AAAI 2024 · similarity 0.89)
- Multi-Task Dense Prediction via Mixture of Low-Rank Experts (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.