Research map / Vision-language and multimodal / CLIP and zero-shot
Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how differently-architected CLIP-trained vision backbones (ViTs and ResNets) differ and shows that adaptively ensembling/cascading them improves accuracy over single backbones at higher compute.
Problem
CLIP-trained vision backbones of different architectures (ViT vs ResNet) are used as general vision solutions, but their differences and complementarity are not well understood; the question is whether combining them can improve performance.
Approach
Treat diverse CLIP backbones as complementary experts: combine them via ensembling (logit averaging, voting) or adaptive cascading that routes inputs through backbones by confidence, trading compute against accuracy.
Key results
- Ensembling diverse CLIP backbones (ViT-B-32, B-16, L-14, L-14-336 plus ResNets) reaches about 77.6% average accuracy at ~1000 GFLOPs, versus 67.5% for the best single zero-shot backbone (ViT-L-14-336) at ~395 GFLOPs.
- Cascading backbones reaches 76.9% average accuracy at 714 GFLOPs, competitive with the full ensemble at lower compute.
- Simple logit averaging across backbones gives 69.0% at 1000 GFLOPs, below the full ensemble but above voting-based combinations (VoteOne 68.2%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an ensemble of combined backbones instead of LogitAvg | Combined-backbone CLIP prediction · Average accuracy (combined backbones) | accuracy: 58.9 -> 62.01; 62.1 -> 68.24; 66.8 -> 75.55; 69.0 -> 77.58 | Helped |
| Use an ensemble of combined backbones instead of VoteOne | Combined-backbone CLIP prediction · Average accuracy (combined backbones) | accuracy: 58.60 -> 62.01; 61.7 -> 68.24; 65.7 -> 75.55; 68.2 -> 77.58 | Helped |
| Use an ensemble of combined backbones instead of a cascade | Combined-backbone CLIP prediction · Average accuracy (combined backbones) | accuracy: 62.01 -> 62.01; 68.13 -> 68.24; 74.95 -> 75.55; 76.90 -> 77.58 | Mixed |
| Add more combined backbones | Ensemble · Average accuracy (combined backbones) | accuracy: 62.01 -> 68.24 -> 75.55 -> 77.58 | Helped |
Surprising findings
- Adaptive cascades of diverse backbones can match full ensembles at lower compute, suggesting backbone diversity is exploitable beyond simple averaging.
Limitations
- Only the abstract fragment and a figure are available; full method details and evaluation datasets are not visible in the provided text.
Experimental setup
Metrics: average accuracy GFLOPs
Baselines: ViT-B-32 zero-shot ViT-L-14 zero-shot RN50x64 zero-shot LogitAvg VoteOne
Builds on: CLIP
Closest papers by meaning
- Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations (CVPR 2024 · similarity 0.90)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.89)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.89)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.88)
- What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights (NeurIPS 2024 · similarity 0.88)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.