Research map / Architectures and efficiency / Mixture of experts

DISTMOE: Rehearsal-free Routing in Mixture-of-Experts for Distributed Instruction Tuning

NeurIPS 2026 · Architectures and efficiency › Mixture of experts · approach: Efficient MoE language model systems

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DistMoE, a rehearsal-free MoE framework that trains client-specific private FFN experts on top of a shared public MLLM and recalibrates routers/adapters with an isotropic Gaussian residual loss for distributed visual instruction tuning.

Problem

Adapting MLLMs to diverse visual-language domains when data is distributed across private clients, without centralized data access or rehearsal data for router training, and with merged experts whose outputs and routing scores are not comparable.

Approach

Each client trains a private FFN expert initialized from a public FFN, then Stage-II freezes experts and recalibrates public/private routers and linear adapters on local data plus a small public subset, using an Epps-Pulley isotropic Gaussian loss on client residuals relative to the public expert so that experts from different clients have comparable scales.

Key results

• With Qwen-1.8B, DistMoE reaches 70.27 VQAv2 and 55.35 GQA, versus 70.01 and 54.26 for rehearsal-free FlexOlmo, and 1188.56 MME perception versus 1176.35 for FlexOlmo.
• On POPE with StableLM-1.6B, DistMoE reaches 84.98 random accuracy versus 77.84 for FlexOlmo.
• Ablation shows Stage-II without adapter, isotropic loss, or client bias yields no gain over Stage-I (e.g., GQA 54.21 vs 54.26), while the full configuration reaches 55.35 GQA.

Surprising findings

• With Phi-2.7B, DistMoE-LoRA performs essentially identically to the public model on most QA tasks (e.g., VQAv2 42.89 vs public 42.87), suggesting limited gains from the LoRA variant there.
• DistMoE with Phi-2.7B underperforms on SQA (66.34) relative to the public model (69.26) and on LLaVA-Bench (55.90 vs 73.30 public).

Limitations

• Stage-II requires a small public calibration set, which may not be available for open-weight models.
• Centralized or rehearsal-based MoE training still performs better on some benchmarks.
• Results on LLaVA-Bench and MM-Vet are mixed; the full DistMoE underperforms on some instruction-following metrics relative to LoRA variant and public model.

Experimental setup

Datasets: COCO GQA OCRVQA TextVQA Visual Genome SViT-157k LVIS-220k LRV-331k MIMIC-IT-256k LLaVA 1.5-558k

Benchmarks: VQAv2 GQA ScienceQA-IMG TextVQA POPE MME SEED-Bench LLaVA-Bench (in-the-Wild) MM-Vet

Metrics: accuracy F1 perception score overall score

Baselines: Model Soup BTM BTX FlexOlmo FlexOlmo-LoRA FlexOlmo-RT MoE-LLaVA Qwen-VL LLaVA-Phi

Builds on: MoE-LLaVA FlexOlmo LeJEPA Branch-Train-Mix LLaVA

Closest papers by meaning

• Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.89)
• Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.89)
• Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning (ICML 2025 · similarity 0.88)
• Pilot: Building the Federated Multimodal Instruction Tuning Framework (AAAI 2025 · similarity 0.87)
• LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
• FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models (NeurIPS 2026 · similarity 0.87)