Research map / Vision-language and multimodal / Multimodal LLMs

Mordal: Automated Pretrained Model Selection for Vision Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Mordal automatically selects the best pretrained vision encoder and LLM combination for a VLM on a user task, using CKA-based candidate clustering, successive halving, and observational scaling-law prediction to cut search cost 8.9-11.6x versus grid search.

Problem

Choosing pretrained vision encoders and LLMs for a VLM on a downstream task is costly because exhaustive grid search requires fully training many candidates (over 100 GPU hours each), and existing transferability-based model selection methods fail for VLMs that need vision-text alignment.

Approach

Cluster VLM candidates by CKA representation similarity (two-step: vision encoders, then LLMs), evaluate one representative per cluster with successive halving, and predict full-data performance from a log-linear observational scaling law fitted on small alignment-data subsets.

Key results

• Mordal finds the top-1 VLM candidate on six of seven tasks while using 8.9x-11.6x fewer GPU hours than grid search (5439 GPU hours for 49 candidates).
• Mordal achieves weighted Kendall's tau of 0.96 on ScienceQA versus 0.77 for the best baseline (EMMS), and averages about 69% higher weighted tau than baselines.
• On the small-model setting (2-3B LLMs), Mordal matches grid-search top-1 on GQA, ChartQA and ScienceQA with 6.7x-8.8x savings, though with lower Kendall's tau.

Surprising findings

• Training-free baselines (EMMS, LogME, LEEP, NLEEP) on unaligned VLMs yield low rank correlation, while Mordal's CKA clustering without warmup remains relatively robust.
• Off-the-shelf VLMs such as LLaVA-1.5 often do not contain the best pretrained components; the best combination differs by task.

Limitations

• Does not guarantee top-1 selection; poor clustering or early stopping can eliminate strong candidates (missed top-1 on ChartQA).
• Scaling prediction is less accurate for smaller models, lowering Kendall's tau in the 2-3B setting.
• Sensitive to hyperparameters such as clustering thresholds and top-k values.

Experimental setup

Datasets: LLaVA-1.5-Instruction GQA VizWiz ChartQA DocVQA ScienceQA AI2D MMMU

Benchmarks: GQA VizWiz ChartQA DocVQA ScienceQA AI2D MMMU

Metrics: top-1 score GPU hours weighted Kendall's tau

Baselines: Grid search EMMS LogME LEEP NLEEP CLIP-Vicuna (LLaVA-1.5)

Builds on: LLaVA-1.5 Successive Halving Algorithm EMMS Cornstarch Centered kernel alignment

Closest papers by meaning

• What matters when building vision-language models? (NeurIPS 2024 · similarity 0.92)
• Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.91)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
• MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.90)
• Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment (CVPR 2025 · similarity 0.90)
• MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs (ICLR 2026 · similarity 0.90)