Research map / Vision-language and multimodal / CLIP and zero-shot
Bridge the Modality and Capability Gaps in Vision-Language Model Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SWAB, which uses optimal-transport class relevance between open-source and target datasets to bridge the modality gap and capability gap for language-only selection of pre-trained VLMs from a zoo.
Problem
Selecting the best pre-trained vision-language model for a zero-shot image classification target task using only the target's text (class names and generated captions), without target images; existing text-proxy methods suffer from a modality gap between image and text embeddings and a capability ga
Approach
Use optimal transport over MPNet class-name similarity to build a bridge matrix that transfers class-level modality gap vectors and class-level VLM rankings from open-source datasets to the target dataset, then ensemble the resulting ranking predictions via weighted Borda count.
Key results
- SWAB achieves R5 + tau of 0.822 versus 0.716 for ModelGPT (about 14.8% improvement), with R5 0.504 and tau 0.318.
- Average Rank over open-source classes (R5 0.443) underperforms ImageNet benchmarking (INB, R5 0.443 with tau 0.267 vs 0.246), supporting the capability gap claim; mean max-min ranking difference across datasets is 34.86 of 43 VLMs.
- Ablation: SWAB-C (capability gap only) gets R5+tau 0.783, SWAB-M (modality gap only) 0.790, full SWAB 0.822.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SWAB instead of ModelGPT | LOVM selection · LOVM Benchmark | R5 + tau: 0.716 ± 0.011 -> 0.822 ± 0.002 | Helped |
| add Capability Gap bridging to Modality Gap bridging (ablation) | SWAB-M · LOVM Benchmark | R5 + tau: 0.790 ± 0.017 -> 0.822 ± 0.002 | Helped |
| add Modality Gap bridging to Capability Gap bridging (ablation) | SWAB-C · LOVM Benchmark | R5 + tau: 0.783 ± 0.019 -> 0.822 ± 0.002 | Helped |
| replace average class ranking with OT-weighted class ranking | LOVM selection · LOVM Benchmark | R5: 0.443 -> 0.513 | Helped |
| replace average class ranking with OT-weighted class ranking | LOVM selection · LOVM Benchmark | tau: 0.246 -> 0.217 | Hurt |
| bridge the Modality Gap in text-derived rankings | SWAB modality-bridging ranking · LOVM Benchmark | R5 + tau: 0.277 -> 0.451 | Helped |
| use class-level instead of dataset-level mean gap vectors (ablation) | SWAB-M · LOVM Benchmark | R5 + tau: 0.747 -> 0.790 | Helped |
Surprising findings
- Average ranking over open-source classes performs worse than using ImageNet ranking alone, despite using more data.
- Partial optimal transport (mass=0.9) worked better than standard OT for bridging the capability gap, unlike for the modality gap.
Limitations
- Relies on ChatGPT-generated auxiliary texts and on ModelGPT's pipeline, so quality depends on generated text.
- Requires labeled open-source datasets with images and class names to compute statistics.
- Some design choices (e.g., partial OT mass 0.9, alpha=0.5, lambda=0.5) are set by hand and their sensitivity is not fully analyzed.
Experimental setup
Datasets: ImageNet SUN397 Country211 Stanford Cars Flowers102 CIFAR100 DTD RESISC45 GTSRB Oxford Pets VOC2007 STL10
Benchmarks: LOVM Benchmark (43 VLMs x 23 datasets)
Metrics: Top-5 Recall (R5) Kendall's tau R5 + tau
Baselines: H-Score NCE LEEP LogME INB (ImageNet benchmarking) Average Rank ModelGPT
Builds on: ModelGPT LOVM benchmark H-Score LogME optimal transport
Closest papers by meaning
- Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance (CVPR 2026 · similarity 0.91)
- Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.91)
- Why are Visually-Grounded Language Models Bad at Image Classification? (NeurIPS 2024 · similarity 0.90)
- The Neglected Tails in Vision-Language Models (CVPR 2024 · similarity 0.90)
- Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · similarity 0.90)
- An Empirical Study Into What Matters for Calibrating Vision-Language Models (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.