Research map / Vision-language and multimodal / Multimodal LLMs

Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Gromov-Wasserstein distance between vision-encoder and LLM representation spaces as a training-free, inference-only score for selecting vision encoders for VLM training, validated on 18 encoders and 60+ LLaVA-style training runs.

Problem

Choosing which pre-trained vision encoder yields the best VLM after visual instruction tuning is costly; common heuristics (largest size, highest zero-shot accuracy) fail to predict final VLM performance.

Approach

Measure the structural (within-modality pairwise distance) similarity between vision-encoder and LLM text representations via a Gromov-Wasserstein distance with median-ratio scale matching, which needs no common embedding space, and argue theoretically that a small GW-infinity distance bounds the Lipschitz constant of the optimal cross-modal mapping.

Key results

• On Qwen-2.5-7B-Instruct, the GW-selected encoder achieves a 66.43 average in the Large group, beating zero-shot accuracy (64.63) and RSA (62.92) and matching the optimum.
• Correlation with final VLM performance over 14 encoders: GW |r|=0.66, |rho|=0.53, R^2=0.43, versus best baseline accuracy |r|=0.46 and RSA |r|=0.48.
• GW estimation takes about 1 minute on one GH200 using 1,000 pairs, versus about 68 GPU-hours for full VLM training.

Surprising findings

• Zero-shot accuracy and model size often fail to identify the best vision encoder for VLM training.
• MutualNN shows a negative correlation with final VLM performance despite Pearson |r| above 0.5, which the authors attribute to a spurious correlation.

Limitations

• Only 18 vision encoders and two LLMs were tested, and the authors call the work an initial step.
• Scope is limited to image and text modalities.
• Zero-shot accuracy baseline and MLCD were excluded for lacking data, so correlation analysis uses only 14 encoders.

Experimental setup

Datasets: LCS-558k (LAION/CC/SBU re-captioned by BLIP) CC3M-595k LLaVA-1.5 665K instruction data

Benchmarks: TextVQA ScienceQA GQA AI2D MMMU MME SEED-I POPE RealWorldQA

Metrics: average score across 9 benchmarks Pearson correlation Spearman correlation R-squared

Baselines: Zero-shot accuracy RSA CCA MutualNN Worst model

Builds on: LLaVA-1.5 Gromov-Wasserstein distance Platonic Representation Hypothesis

Closest papers by meaning

• Bridge the Modality and Capability Gaps in Vision-Language Model Selection (NeurIPS 2024 · similarity 0.91)
• Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.91)
• Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs (NeurIPS 2026 · similarity 0.91)
• Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.90)
• What matters when building vision-language models? (NeurIPS 2024 · similarity 0.90)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)