Research map / Vision-language and multimodal / Multimodal LLMs
MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoME, a mixture of multimodal experts combining a vision-expert mixture (MoVE, with adaptive deformable transformation over CLIP, DINOv2, Pix2Struct features) and a language-expert mixture (MoLE, sparse adapters in FFN layers) to reduce task interference in generalist MLLMs.
Problem
Generalist MLLMs trained on mixed vision-language tasks underperform specialist models due to task interference, and prior MoE-for-MLLM work only addresses the language side while ignoring visual differences.
Approach
Apply mixture-of-experts on both the vision side (routing and aggregating features from multiple vision encoders after an adaptive deformable transformation that aligns their sequence lengths) and the language side (instance-level top-1 routed low-rank adapters parallel to each LLM FFN), with routing conditioned on a Sentence-BERT instruction embedding.
Key results
- MoVE with ADT and router reaches 69.39 average vs 56.52 for the best single CLIP encoder (Table 1), and MoVE gains about 12.87 points average over baselines per the introduction.
- MoME achieves 130.8 CIDEr on TextCaps and 94.6 on Flickr30K, outperforming listed baselines such as LION (108.8) and MoCLE (81.9) in Table 3.
- MoLE-I (instance-level sparse router) is the best language-expert configuration, improving the single-adapter baseline from 57.32 to 58.94 average with CLIP-only encoder.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MoME rather than the strongest listed baseline | DocVQA · DocVQA test | ANLS: 47.1 -> 50.8 | Helped |
| use MoME rather than the strongest listed baseline | ChartQA · ChartQA test | Relax Accuracy: 46.9 -> 57.2 | Helped |
| use MoME rather than the strongest listed baseline | TextCaps · TextCaps val | CIDEr: 108.8 -> 130.8 | Helped |
| use MoME rather than the strongest listed baseline | TextVQA · TextVQA val | Accuracy: 60.2 -> 53.2 | Hurt |
| use MoME rather than the strongest listed baseline | Flickr30K · Flickr30K karpathy-test | CIDEr: 87.4 -> 94.6 | Helped |
| use MoME rather than the strongest listed baseline | IconQA · IconQA test | Accuracy: 54.89 -> 61.4 | Helped |
| use MoME rather than the strongest listed baseline | VSR · VSR val | Accuracy: 73.8 -> 61.9 | Hurt |
| use MoME rather than the strongest listed baseline | GQA · GQA test-dev | Accuracy: 62.0 -> 59.7 | Hurt |
| replace single vision encoders with MoVE using ADT and instruction-based Router | MoVE · General, REC, REG, Document (average) | Avg.: 56.52 -> 69.39 | Helped |
| add ADT instead of AvgPool for vision-feature transformation (ablation) | MoVE with Addition aggregation · General, REC, REG, Document (average) | Avg.: 58.91 -> 62.88 | Helped |
| replace Addition with an instruction-based Router for feature aggregation (ablation) | MoVE with ADT · General, REC, REG, Document (average) | Avg.: 62.88 -> 69.39 | Helped |
| replace a single adapter with instance-level sentence-embedding routing (ablation) | MoLE with CLIP · General, REC, REG, Document (average) | Avg.: 57.32 -> 58.94 | Helped |
Surprising findings
- Simple addition-based fusion of multiple vision encoders gives only a small average gain and some sub-tasks decline, while learned routing with ADT is needed for gains.
- Gumbel-softmax noise and auxiliary load-balancing loss both hurt MoLE performance compared with plain instance-level routing.
Limitations
- Experiments limited by compute: not extended to more data or modalities
- Relies on a single 7B LLM (Vicuna-v1.5) and three fixed vision encoders
- Some results trail baselines (e.g., GQA, TextVQA, VSR in Table 3)
Experimental setup
Datasets: COCOCap Flickr30K NoCaps OKVQA AOKVQA GQA VSR IconQA RefCOCO RefCOCO+ RefCOCOg ChartQA
Benchmarks: COCOCap Flickr30K NoCaps OKVQA AOKVQA GQA VSR IconQA RefCOCO RefCOCO+ RefCOCOg ChartQA
Metrics: CIDEr Accuracy ANLS F1 Score Relaxed Accuracy
Baselines: Shikra-7B Ferret-7B InstructBLIP LLaVA-v1.5 LION DocPedia MoE-LLaVA MixLoRA MoCLE LLaVA-MoLE
Builds on: LLaVA Deformable DETR MoE-LLaVA InstructBLIP Vicuna
Closest papers by meaning
- CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts (NeurIPS 2024 · similarity 0.89)
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.89)
- MoVA: Adapting Mixture of Vision Experts to Multimodal Context (NeurIPS 2024 · similarity 0.87)
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (NeurIPS 2025 · similarity 0.87)
- Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning (ICML 2025 · similarity 0.87)
- Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.