Research map / Vision-language and multimodal / Multimodal LLMs
CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CuMo, which adds co-upcycled Top-K sparse MoE blocks to the CLIP vision encoder and MLP connector of a multimodal LLM, trained in three stages on open-source data, reaching state-of-the-art results within model-size groups.
Problem
Scaling multimodal LLMs via more data or larger LLMs is computationally expensive, and vision-side scaling is comparatively unexplored; the paper asks how to add capacity on the vision side without greatly increasing activated parameters.
Approach
Replace MLP blocks in the CLIP vision encoder and the vision-language connector with Top-K sparsely-gated MoE blocks, initialize each expert from the pre-trained MLP (co-upcycling) during visual instruction tuning, and stabilize routing with load-balance and router z-losses, with a pre-finetuning warm-up stage.
Key results
- CuMo Mistral-7B (7.8B activated) outperforms 7B-group models such as LLaVA-NeXT-Mistral-7B and MM1-7B on several benchmarks, e.g. MMBench-EN 73.0 and MM-Vet 51.0.
- CuMo Mixtral-8x7B (13.5B activated) reaches SEED-IMG 73.2 and MMMU 45.0, competitive with SPHINX-MoE, MM1-7B-MoE and Mini-Gemini in the MoE group.
- Under limited data (LLaVA-558K and LLaVA-665K only), CuMo Mistral-7B reaches comparable performance to LLaVA-v1.5 Vicuna-13B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Upcycle MLP experts from pre-trained MLP (instead of training MoE from scratch) in MLP connector (ablation) | MLP-MoE connector, Top 2-in-4 · SQA-IMG | accuracy: 68.1 -> 73.7 | Helped |
| Add bzloss (load balance + router z-loss) to upcycled MLP-MoE (ablation) | MLP-MoE connector · MM-Vet | score: 32.3 -> 33.1 | Helped |
| Top 2-in-8 router instead of Top 2-in-4 with upcycling and bzloss (ablation) | MLP-MoE connector · SQA-IMG | accuracy: 73.5 -> 73.4 (slight drop in SQA, MM-Vet 33.1 -> 32.4) | Hurt |
| Upcycling MoE in CLIP vision encoder (Top 2-in-4) with bzloss, after unfreezing CLIP (ablation) | CLIP vision encoder · TextVQA | accuracy: 57.4 -> 59.7 (MLP-MoE baseline to CLIP-MoE) | Helped |
| Train CLIP MoE blocks from scratch instead of upcycling | CLIP vision encoder · training convergence | convergence | Hurt |
| Upcycle LLM MLPs into MoE (Mistral 4x7B / 8x7B) instead of pre-trained Mixtral 8x7B (ablation) | LLM · TextVQA | accuracy: upcycled 8x7B 56.4 vs Mixtral 8x7B 60.6 | Hurt |
| Use pre-trained Mixtral 8x7B instead of upcycled Mistral MoE in LLM (ablation) | LLM · MM-Vet | score: 35.7 -> 40.0 | Helped |
| Multi-resolution CLIP input (3x + 1x) channel-wise concatenation (ablation) | CLIP vision encoder · MM-Vet | score: 34.3 -> 37.0 | Helped |
| Adding 2x multi-resolution input alone or with 3x (ablation) | CLIP vision encoder · SQA-IMG | accuracy: 71.7 -> 71.7 (1x+2x); 72.2 (all three) | No clear effect |
| Add pre-finetuning stage with ALLaVA caption data (all params unfrozen) (ablation) | full model before visual instruction tuning · TextVQA | accuracy: 59.3 -> 62.8 | Helped |
| Pre-finetuning with ShareGPT4V captions instead of ALLaVA (ablation) | full model before visual instruction tuning · TextVQA | accuracy: 59.3 -> 61.7 (ALLaVA gives 62.8) | Helped |
| CuMo Mistral-7B (MoE in CLIP + MLP, with bzloss) vs LLaVA-v1.5 Vicuna-13B under limited data | CuMo Mistral-7B · limited training data (LLaVA-558K/665K) vs LLaVA-v1.5-13B | MMBench EN: 69.6 vs 67.7 | Mixed |
Surprising findings
- Co-upcycled Mistral-based MoE LLMs gave only marginal gains over the dense baseline, so LLM upcycling was dropped in favor of the pre-trained Mixtral 8x7B.
- Training MoE blocks from scratch failed to converge, even with lower learning rates, in both the MLP connector and CLIP.
Limitations
- Upcycled experts add parameters without new knowledge; the paper notes the LLM is the dominant parameter share and vision scaling remains limited.
- Hallucinations persist in dialogue examples.
- Top-2-in-8 routing in the connector slightly degraded performance, possibly due to limited instruction data.
Experimental setup
Datasets: LLaVA-558K ALLaVA-Caption LLaVA-665K ShareGPT4V LAION-GPT-V DocVQA ChartQA AI2D InfoVQA SynDog-EN ALLaVA LIMA
Benchmarks: SQA-IMG TextVQA GQA POPE MME MMBench MM-Vet VQAv2 LLaVA-Wild SEED-IMG MMMU MathVista
Metrics: accuracy score
Baselines: LLaVA-v1.5 LLaVA-NeXT Mini-Gemini MM1 SPHINX-MoE VILA InstructBLIP Qwen-VL-Chat GPT4V Gemini 1.5 Pro
Builds on: sparse upcycling LLaVA v1.5 CLIP Mixtral 8x7B S^2 multi-resolution
Closest papers by meaning
- MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models (NeurIPS 2024 · similarity 0.89)
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.89)
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.89)
- Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight (NeurIPS 2024 · similarity 0.88)
- Scaling Laws for Multimodal Data Mixtures (NeurIPS 2026 · similarity 0.88)
- Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.