Research map / Vision-language and multimodal / CLIP and zero-shot
Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes three ensemble strategies (zero-shot confidence-aware ensemble, training-free greedy-search ensemble, and tuning ensemble with a sample-aware weight generator) that combine multiple CLIP vision-language models to improve generalization, including on base-to-new and cross-dataset benchmarks.
Problem
Single fine-tuned CLIP-based VLMs (e.g., CLIP-ViT-B/16) show saturated open-world generalization, so gains from intricate single-model prompt-learning methods are limited.
Approach
Ensemble multiple pre-trained CLIP models, including weaker ones, that make different errors: a confidence-aware logit weighting that keeps ViT-B/16 dominant (zero-shot), a greedy-searched weight set (training-free), and a sample-aware MLP weight generator over image features (tuning).
Key results
- Zero-shot ZS_En improves average accuracy over CLIP-ViT-B/16 by 2.61% across 11 datasets (67.88% vs 65.27%) and ImageNet from 66.72% to 70.66%.
- Tuning ensemble PromptSRC + T_En reaches 81.11 HM on base-to-new average, versus 79.54 for PromptSRC reproduced; CoCoOp + T_En reaches 79.20 HM vs 75.60 for CoCoOp.
- On cross-dataset evaluation (ImageNet source), T_En achieves 67.46% average over 10 target datasets, the best among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Ensemble CLIP-ViT-B/16 with weaker CLIP models using confidence-aware weighting | CLIP-ViT-B/16 · 11-dataset average | accuracy: +2.61 | Helped |
| Add RN50, RN101, and ViT-B/32 to CLIP-ViT-B/16 (ablation) | CLIP-ViT-B/16 · ImageNet | accuracy: 66.72 -> 68.76 | Helped |
| Preserve ViT-B/16 dominance while applying CAW to the other three models (ablation) | Four-model CLIP ensemble · ImageNet | accuracy: 68.76 -> 70.66 | Helped |
| Use three output weights instead of four in SWIG (ablation) | T_En · ImageNet base-to-new | HM: 74.30 -> 74.20 | Hurt |
| Use concatenated logits instead of image features as SWIG input (ablation) | T_En · ImageNet base-to-new | HM: 74.30 -> 73.44 | Hurt |
| Vary SWIG downsampling scale from 32 to 4 (ablation) | T_En · ImageNet base-to-new | HM: 74.30 -> 74.21 | No clear effect |
| Add T_En to PromptSRC | PromptSRC · Base-to-new average | HM: 79.97 -> 81.11 | Helped |
| Add T_En to CoCoOp | CoCoOp · Domain generalization average | accuracy: +1.75% | Helped |
| Use four random augmentations with a single ViT-B/16 instead of ZS_En (ablation) | CLIP-ViT-B/16 · ImageNet | accuracy: 70.66% -> 66.81% | Hurt |
| Use ZS_En instead of mean ensembling | BLIP and BLIP2 · ImageNet | accuracy: 51.39 -> 56.38 | Helped |
| Use ZS_En instead of mean ensembling | BLIP, BLIP2, and CLIP-ViT-B/32 · ImageNet | accuracy: 65.44 -> 67.88 | Helped |
Surprising findings
- Weaker CLIP models (e.g., RN50) sometimes outperform stronger ones (e.g., RN101) on several datasets, and ResNet-based CLIP scaling does not consistently help.
- Ensembling four augmented views of a single ViT-B/16 yields only 66.81% on ImageNet, far below the multi-model ensemble.
Limitations
- Ensemble adds inference cost from running multiple encoders
- Exploration is preliminary and covers only a limited set of models and ensemble strategies, as the authors acknowledge
- Domain generalization gains for CLIP + T_En are smaller than state-of-the-art prompt-learning methods
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNet-V2
Metrics: accuracy harmonic mean (HM) base accuracy new accuracy
Baselines: CLIP CoOp CoCoOp MaPLe PromptSRC ProDA Tip-Adapter + SHIP Mean ensemble
Builds on: CLIP CoCoOp PromptSRC MaPLe Tip-Adapter
Closest papers by meaning
- Adaptive Parameter Selection for Tuning Vision-Language Models (CVPR 2025 · similarity 0.90)
- Weak Distribution Detectors Lead to Stronger Generalizability of Vision-Language Prompt Tuning (AAAI 2024 · similarity 0.89)
- Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.89)
- Realistic Test-Time Adaptation of Vision-Language Models (CVPR 2025 · similarity 0.89)
- Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models (ICLR 2024 · similarity 0.89)
- Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.