Research map / Vision-language and multimodal / Multimodal LLMs

Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Retrains all 31 non-empty subsets of five vision encoders in Eagle-X5 on Cambrian-1's 16 benchmarks, comparing training-time removal against inference-time masking and proposing a Capacity–Necessity decomposition plus a pre-projector effective-rank analysis for encoder selection.

Problem

Inference-time masking of encoders in multi-encoder LVLMs does not reflect joint training, so encoder roles and good compact encoder pairs are hard to identify before training.

Approach

Replace inference-time masking with retraining every encoder subset under a unified recipe, decompose each encoder's value into Capacity (singleton score relative to full pool) and Necessity (leave-one-out drop), and link pair quality to pre-projector effective rank changes under joint training.

Key results

• Training-time removal and inference-time masking rank different encoders first (EVA-02 under IM vs ConvNeXt under TR); rank correlation Spearman 0.82 overall.
• Best retrained pair CLIP+ConvNeXt scores 62.28 average, closing 97% of CLIP-to-full gap, versus 62.54 for the full five-encoder pool.
• Singleton pre-projector effective rank correlates with singleton score (Pearson r=0.89); pool-level score vs summed rank r=0.67.

Surprising findings

• The two highest-Capacity encoders (ConvNeXt and EVA-02) do not form the best pair; CLIP+ConvNeXt wins instead.
• Knowledge benchmarks are dominated by the LLM prior, with even SAM reaching near full-pool scores on Knowledge.

Limitations

• Findings calibrated to channel-concatenation fusion, the Vicuna-7B decoder, and a single five-encoder pool.
• Each subset is trained once, so per-family differences are directional rather than statistically significant.
• Rank-expansion heuristic is descriptive for the Eagle-X5 pool and not a derived selection rule for unseen pools.

Experimental setup

Datasets: Cambrian-1 suite

Benchmarks: MME-Perception MMBench-EN SeedBench GQA ScienceQA MMMU MathVista AI2D OCRBench ChartQA TextVQA DocVQA

Metrics: unweighted 16-task average score Spearman rank correlation Capacity Necessity effective rank CKA

Baselines: Inference-time Masking (CUR, zero-input) Best-at-k subsets Eagle-X5 full pool

Builds on: Eagle Cambrian-1 Prismatic VLMs Conditional Utilization Rate

Closest papers by meaning

• Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders (ICLR 2026 · similarity 0.93)
• Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.92)
• Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.91)
• Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance (CVPR 2026 · similarity 0.91)
• Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.89)
• MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders (CVPR 2025 · similarity 0.89)