Research map / Vision-language and multimodal / Multimodal LLMs

Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMs

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematic study of fusion design for mixture-of-vision-encoder MLLMs, yielding the LEO model (InternViT + SAM with dynamic tiling, tile-level interleaving, and post-adaptation projectors) that beats prior MoVE models on most of 11 benchmarks.

Problem

How to effectively combine multiple vision encoders' tokens in MLLMs while scaling to high-resolution inputs; prior MoVE fusion strategies were studied in isolation without systematic comparison of tiling, merging granularity, and fusion timing.

Approach

Controlled empirical study showing that dynamic tiling with global thumbnail, tile-level sequence interleaving, and post-adaptation fusion with per-encoder MLP projectors together form a lightweight MoVE recipe; instantiated as LEO with InternViT-300M and SAM-L, plus a driving-domain adaptation.

Key results

• LEO reaches 80.1 on DocVQA, 71.0 on ChartQA and 78.5 on ScienceQA, beating the best listed MoVE baselines on the majority of the 11 benchmarks (leads on 7 of 11).
• Post-adaptation fusion improves the 8-benchmark average over pre-adaptation by about 2.9 points on average across encoder pairs (e.g., I+SAM 64.9 to 67.5).
• On LingoQA validation, LEO (2 frames) reaches Lingo-J 61.0 and METEOR 35.44, outperforming the open-source baselines on all four metrics.

Surprising findings

• Freezing both vision encoders during SFT gives the best result (67.5), contrary to Eagle, which found unfreezing necessary.
• Cross-attention merging underperforms simple sequence interleaving and appending despite more parameters.

Limitations

• Processing limited to at most six image patches plus global context due to LLM context length and compute.
• Relies on a fixed set of two vision encoders and a single 7B LLM, limiting generality.

Experimental setup

Datasets: LLaVA-595K Eagle SFT data (~1M, as in baseline) Eagle-1.8M LingoQA DocVQA TextVQA ChartQA AI2D GQA VizWiz VQA v2 MMMU

Benchmarks: TextVQA GQA ChartQA VizWiz MMBench MM-Vet DocVQA POPE SEED ScienceQA MathVista LingoQA

Metrics: accuracy average score Lingo-Judge BLEU METEOR CIDEr latency FLOPs

Baselines: Eagle-X2 Eagle-X3 Brave-X5 DeepSeek-VL LLaVA-HR Mini-Gemini MouSi-X3 SPHINX RADIO MoVE-KD-v1.1 LingoQA InternVL

Builds on: LLaVA Eagle InternVL SAM MoVE-KD

Closest papers by meaning

• Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders (ICLR 2025 · similarity 0.91)
• MoVA: Adapting Mixture of Vision Experts to Multimodal Context (NeurIPS 2024 · similarity 0.90)
• Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders (ICLR 2026 · similarity 0.89)
• Dense Connector for MLLMs (NeurIPS 2024 · similarity 0.89)
• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.89)
• LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.89)