Research map / Vision-language and multimodal / Cross-modal learning
Uni-MoE: Scaling Unified Multimodal LLMs With Mixture of Experts
Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, Min Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| activate top-2 experts instead of top-1 (ablation) | Uni-MoE · RACE-Audio middle | accuracy: 45.13% -> 46.31% | Improved |
| use 4 experts instead of 1 (ablation) | Uni-MoE · OK-VQA | accuracy: 56.12% -> 63.02% | Improved |
| use 4 experts instead of 2 (ablation) | Uni-MoE · RACE-Audio middle | accuracy: 44.64% -> 46.31% | Improved |
| add more identical experts (6 instead of 1) to pure MoE (ablation) | Pure MoE · ActivityNet-QA | accuracy: 42.5% -> 43.2% | No clear effect |
| apply MoE to all layers instead of interval layout (ablation) | Uni-MoE · ActivityNet-QA | accuracy: 43.3% -> 42.8% | Hurt |
| apply MoE only to second half of layers instead of all layers (ablation) | Uni-MoE · RACE-Audio high | accuracy: 43.71% -> 52.69% | Improved |
| pre-tune modality-specific experts (tri-phase training) instead of identical LLaVA experts (ablation) | Uni-MoE · avg of 5 benchmarks | average accuracy: 47.5% -> 49.2% | Improved |
| add auxiliary balancing loss to mixture(4) experts (ablation) | Uni-MoE · avg of 5 benchmarks | average accuracy: 49.2% -> 48.5% | Hurt |
| add auxiliary balancing loss to pure(4) experts (ablation) | Pure MoE · avg of 5 benchmarks | average accuracy: 47.5% -> 48.4% | Improved |
| use pure(4) MoE instead of single dense expert (LLaVA experts) (ablation) | Pure MoE · avg of 5 benchmarks | average accuracy: 48.2% -> 47.5% | No clear effect |
| use pure(4) MoE instead of single dense expert (LLaMA experts) (ablation) | Pure MoE · avg of 5 benchmarks | average accuracy: 46.5% -> 47.3% | Improved |
| expand image-text data from LLaVA-150k to LLaVA-v1.5-665k | Uni-MoE (4E) · MMBench | accuracy: 69.82% -> 73.2% | Improved |
Closest papers by meaning
- CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts (NeurIPS 2024 · similarity 0.91)
- Scaling Laws for Multimodal Data Mixtures (NeurIPS 2026 · similarity 0.90)
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.90)
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.89)
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.89)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
