Research map / Vision-language and multimodal / CLIP and zero-shot
Anchor-based Robust Finetuning of Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Anchor-based Robust Finetuning (ARF), which finetunes CLIP with contrastive losses on BLIP2-generated captions for ImageNet images and CLIP-retrieved CC3M image-text pairs, to preserve both domain-shift and zero-shot generalization.
Problem
Finetuning CLIP on downstream data degrades out-of-distribution generalization, both under domain shift and for categories absent from the finetuning set, because class-name-only text targets are semantically sparse.
Approach
Regularize finetuning with two kinds of rich-semantic image-text anchors: captions generated by a pretrained captioner for finetune images (TCAG), and image-text pairs retrieved from CC3M via CLIP embeddings (ITAR), each aligned with a CLIP contrastive loss alongside the class-prompt loss.
Key results
- ARF reaches 61.3% average domain-shift accuracy on ImageNet variants versus 59.4% for FLYP, with ID accuracy 82.7%.
- On zero-shot recognition after ImageNet finetuning, ARF averages 55.6% OOD accuracy versus 48.6% for FLYP and 35.7% for FT.
- On DomainNet, ARF achieves 65.2% domain-shift average, outperforming CLIP's 64.1% while keeping ID accuracy at 65.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add text-compensated anchors generated by TCAG (ablation) | ARF finetuning on ImageNet · ImageNet domain shift (Avg. OOD) | top-1 accuracy: +1.3% | Helped |
| add retrieved image-text anchors using ITAR (ablation) | ARF finetuning on ImageNet · Zero-shot learning (Avg. OOD) | top-1 accuracy: +5.0% | Helped |
| add both text-compensated and retrieved image-text anchors (ablation) | ARF finetuning on ImageNet · Zero-shot learning (Avg. OOD) | top-1 accuracy: +7.0% | Helped |
| use BLIP2 rather than BLIP to generate captions (ablation) | ARF finetuning on ImageNet · Zero-shot learning (Avg. OOD) | top-1 accuracy: +1.6% | Helped |
| rewrite BLIP2 captions with Vicuna (ablation) | ARF finetuning on ImageNet · Zero-shot learning (Avg. OOD) | top-1 accuracy: +0.9% | Helped |
| rewrite BLIP2 captions with Vicuna (ablation) | ARF finetuning on ImageNet · ImageNet ID and domain shift | top-1 accuracy | No clear effect |
Surprising findings
- Plain full finetuning drops zero-shot accuracy to 35.7% average, well below the original CLIP's 66.6%, and even FLYP loses substantial zero-shot accuracy.
Limitations
- Gains on domain shift over FLYP are modest (about 1.9 points), and ARF's ID accuracy is only comparable to finetuning.
- Zero-shot evaluation is done only after ImageNet finetuning, with a fixed ViT-B/16 backbone.
- Retrieval relies on a CC3M candidate set and the downstream task's images, which adds offline precomputation and dataset dependence.
Experimental setup
Datasets: ImageNet DomainNet CC3M
Benchmarks: ImageNet-V2 ImageNet-Sketch ImageNet-R ImageNet-A ObjectNet DomainNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 SUN397
Metrics: top-1 accuracy
Baselines: LP FT LP-FT FLYP Wise-FT CLIP
Builds on: CLIP FLYP Wise-FT LP-FT BLIP2
Closest papers by meaning
- Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization (ICLR 2024 · similarity 0.91)
- Reproducible Vision-Language Models Meet Concepts Out of Pre-Training (CVPR 2025 · similarity 0.89)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.89)
- Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification (CVPR 2024 · similarity 0.89)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.88)
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.