Research map / Vision-language and multimodal / CLIP and zero-shot
Vision-Language Model Selection and Reuse for Downstream Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Model Label Learning (MLL), which labels VLMs on a WordNet-based semantic graph, then selects and ensembles them for zero-shot downstream tasks, and releases a 49-VLM, 17-dataset selection benchmark.
Problem
Selecting the best pre-trained vision-language model for a specific downstream task is hard because no single VLM dominates all tasks and exhaustively evaluating every candidate is infeasible, especially without access to training data.
Approach
Each VLM is pre-tested once, when submitted, on a WordNet-derived semantic graph of about 9055 nodes to produce a model label; for a new task, class names are matched to graph nodes via LLM-generated caption embeddings, and the top-k models per class are ensembled with entropy-based weights. Labeling is target-task independent, so selection needs no runs on the target data.
Key results
- With a single reused model per class (k=1), the proposed method reaches 0.6620 average accuracy across 17 tasks, versus 0.6434 for INB and 0.6367 for ModelGPT.
- With k=3 ensembles, average accuracy is 0.6639 versus 0.6498 (INB) and 0.6480 (ModelGPT).
- Ablation over alpha shows 0.7 gives the best average (0.6620), and performance is fairly stable across alpha in 0.5 to 0.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MLL to select and reuse one model per class | VLM selection on 17 downstream tasks · 17 downstream tasks, k=1 | accuracy (average): 0.6434 -> 0.6620 | Helped |
| use MLL to select and reuse one model per class | VLM selection on 17 downstream tasks · 17 downstream tasks, k=1 | accuracy (average): 0.6367 -> 0.6620 | Helped |
| use MLL to select and ensemble 3 models per class | VLM selection on 17 downstream tasks · 17 downstream tasks, k=3 | accuracy (average): 0.6498 -> 0.6639 | Helped |
| use MLL to select and ensemble 3 models per class | VLM selection on 17 downstream tasks · 17 downstream tasks, k=3 | accuracy (average): 0.6480 -> 0.6639 | Helped |
| set alpha to 0.5 instead of 0.7 (ablation) | MLL · 17 downstream tasks, k=1 | accuracy (average): 0.6620 -> 0.6569 | Hurt |
| set alpha to 0.6 instead of 0.7 (ablation) | MLL · 17 downstream tasks, k=1 | accuracy (average): 0.6620 -> 0.6571 | Hurt |
| set alpha to 0.8 instead of 0.7 (ablation) | MLL · 17 downstream tasks, k=1 | accuracy (average): 0.6620 -> 0.6600 | Hurt |
| set alpha to 0.9 instead of 0.7 (ablation) | MLL · 17 downstream tasks, k=1 | accuracy (average): 0.6620 -> 0.6590 | Hurt |
| reuse 2 models per class instead of 1 (ablation) | MLL · 17 downstream tasks | accuracy (average): 0.6620 -> 0.6637 | Helped |
| reuse 3 models per class instead of 1 (ablation) | MLL · 17 downstream tasks | accuracy (average): 0.6620 -> 0.6639 | Helped |
| reuse 4 models per class instead of 1 (ablation) | MLL · 17 downstream tasks | accuracy (average): 0.6620 -> 0.6571 | Hurt |
| reuse 5 models per class instead of 1 (ablation) | MLL · 17 downstream tasks | accuracy (average): 0.6620 -> 0.6556 | Hurt |
Surprising findings
- ModelGPT largely tracks the ImageNet baseline in its selections, so it fails when INB fails to pick a good model.
- Ensembling more models does not always help; on weak-hub tasks such as CLEVR-D and on datasets with few classes such as PCam, a single model per class performs better.
Limitations
- The current implementation covers only VLMs and visual classification tasks.
- Selection quality depends on the coverage of the semantic graph and on the caption-based matching between task classes and graph nodes.
Experimental setup
Datasets: ImageNet ImageNet-V2 ImageNet-Sketch ImageNet-A ImageNet-R WordNet
Benchmarks: CIFAR100 Country211 CLEVR-D DTD DMLab Flowers102 MNIST OxfordPet PCam FER2013 Food101 GTSRB
Metrics: accuracy
Baselines: ImageNet Baseline (INB) ModelGPT
Builds on: CLIP ModelGPT Learnware open-clip
Closest papers by meaning
- Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance (CVPR 2026 · similarity 0.91)
- Bridge the Modality and Capability Gaps in Vision-Language Model Selection (NeurIPS 2024 · similarity 0.91)
- Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.91)
- Adaptive Parameter Selection for Tuning Vision-Language Models (CVPR 2025 · similarity 0.91)
- Active Prompt Learning in Vision Language Models (CVPR 2024 · similarity 0.90)
- What matters when building vision-language models? (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.