Research map / Vision-language and multimodal / Multimodal LLMs

MoVA: Adapting Mixture of Vision Experts to Multimodal Context

NeurIPS 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MoVA, an MLLM that routes task-specific vision experts via LLM-based context-aware selection (coarse stage) and fuses their features with a mixture-of-vision-expert adapter (fine stage) on top of a CLIP base encoder.

Problem

Single vision encoders (mainly CLIP) are biased toward certain tasks, and plain concatenation of multiple vision experts degrades performance because irrelevant experts inject biased features.

Approach

A coarse-to-fine scheme: the LLM itself routes which vision experts are relevant to the image and instruction using loss-derived routing annotations, then a MoV-Adapter with per-expert cross-attention and a dynamic gating network fuses only the selected experts into the CLIP features.

Key results

• MoVA-7B reaches 70.4 on MMBench and 1562 on MME perception, outperforming LLaVA-NeXT-7B while using only 576 image tokens versus 2880.
• MoVA-34B reports 81.3 on MMBench and 79.0 on MMBench-CN, surpassing GPT-4V's 75.8 and 73.9 as reported in the paper's table.
• MoVA-7B achieves 64.8 on GQA and 68.3 on ChartQA and 81.3 on DocVQA, with ablations showing random routing and removing routing both degrade text-oriented VQA.

Measured effects reported in the paper

Surprising findings

• Plain fusion of multiple vision experts often underperforms the single best expert on its specialty task, indicating biased information from irrelevant experts.
• An MLP classifier or BERT router fails to generalize to open-world expert routing (0% accuracy), while the LLM router reaches 92.5%.

Limitations

• Hallucination remains an issue inherent to MLLMs.
• Performance depends on the quality of context-relevant vision experts; failure cases of experts can degrade results.
• Routing annotations require running multiple LLaVA models offline to compute loss-based labels.

Experimental setup

Datasets: DataComp-1B ShareGPT4V-PT ALLaVA-4V Objects365 RefCOCO VisualGenome PointQA Flickr30K MMC-Instruction Chart2Text DVQA SciGraphQA

Benchmarks: MME MMBench QBench MathVista MathVerse POPE VQAv2 GQA ScienceQA TextVQA ChartQA DocVQA

Metrics: accuracy Acc@0.5 open/closed accuracy

Baselines: LLaVA-1.5 LLaVA-NeXT SPHINX-2k Qwen-VL LLaVA-Med CogAgent PALI-X-55B UNINEXT-H

Builds on: LLaVA-1.5 CLIP Vicuna Co-DETR

Closest papers by meaning

• Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMs (NeurIPS 2026 · similarity 0.90)
• CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
• Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models (NeurIPS 2025 · similarity 0.89)
• MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders (CVPR 2025 · similarity 0.89)
• MoCHA: Advanced Vision-Language Reasoning with MoE Connector and Hierarchical Group Attention (AAAI 2026 · similarity 0.89)
• ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Experts (ICLR 2025 · similarity 0.89)