Research map / Vision-language and multimodal / CLIP and zero-shot
Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CraFT, a collaborative fine-tuning method for black-box CLIP that combines CMA-ES-optimized prompt generation with a residual MLP prediction refiner trained by gradient descent, linked by a prediction-consistent KL loss.
Problem
Fine-tuning vision-language models when only input prompts and output predictions are accessible (no gradients or parameters), for few-shot classification.
Approach
Split the black-box VLM adaptation into an input-side prompt generator optimized by derivative-free CMA-ES in a random low-dimensional subspace and an output-side residual MLP refiner optimized by AdamW, trained alternately with a collaborative algorithm justified by residual decomposition and a KL prediction-consistency loss.
Key results
- Under 16-shot, CraFT improves average accuracy by 12.45% over zero-shot CLIP and 5.82% over BBT on ResNet-50 CLIP.
- CraFT reaches 75.76% average on 11 datasets with ViT-B/16 CLIP, versus 72.42% for BBT and 65.23% for zero-shot CLIP.
- CraFT uses about 1/80 the user-side memory of CoOp (5.0 MB vs 395.7 MB) and trains in 1h44min vs 2h3min, at 1.62% lower accuracy than CoOp on ImageNet 16-shot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CraFT instead of BBT | CLIP ResNet-50 · 11-dataset few-shot average, 16-shot | accuracy: 65.40 -> 71.22 | Helped |
| Use CraFT instead of BBT | CLIP ResNet-101 · 11-dataset few-shot average, 16-shot | accuracy: 68.33 -> 71.15 | Helped |
| Use CraFT instead of BBT | CLIP ViT-B/32 · 11-dataset few-shot average, 16-shot | accuracy: 68.45 -> 71.31 | Helped |
| Use CraFT instead of BBT | CLIP ViT-B/16 · 11-dataset few-shot average, 16-shot | accuracy: 72.42 -> 75.76 | Helped |
| Use CraFT instead of ZS-CLIP | CLIP ResNet-50 · ImageNet distribution-shift average | accuracy: 40.58 -> 42.32 | Helped |
| Use CraFT instead of CoOp (L=4) | CLIP ViT-B/16 · ImageNet distribution-shift average | accuracy: 59.38 -> 58.80 | Hurt |
| Add visual prompts to CraFT (ablation) | CraFT with CLIP ViT-B/16 · 11-dataset few-shot average, 16-shot | accuracy: 75.76 -> 78.40 | Helped |
| Add prediction refinement to prompt generation (ablation) | CraFT, ResNet-50 CLIP · 11-dataset few-shot average, 16-shot | accuracy: 65.40 -> 68.23 | Helped |
| Add prediction refinement to prompt generation (ablation) | CraFT, ResNet-50 CLIP · 11-dataset few-shot average, 16-shot | accuracy: 65.40 -> 71.22 | Helped |
| Add collaborative training to the prompt and refinement modules (ablation) | CraFT, ResNet-50 CLIP · 11-dataset few-shot average, 16-shot | accuracy: 68.23 -> 71.22 | Helped |
| Add residual connection to refinement MLP (ablation) | CraFT, ResNet-50 CLIP · 11-dataset few-shot average, 1-shot | accuracy: 28.58 -> 59.49 | Helped |
| Use an MLP instead of a linear refinement mapping (ablation) | CraFT, ResNet-50 CLIP · 11-dataset few-shot average, 16-shot | accuracy: 65.88 -> 71.22 | Helped |
Surprising findings
- Without the collaborative training algorithm, combining the prompt generator and refiner performs worse than the refiner alone (68.23 vs 69.29 at 16 shots).
- Gradient-free BBT outperforms gradient-based CoOp and CraFT on OxfordPets and Food101, attributed to instability of gradient-based fine-tuning on fine-grained datasets.
Limitations
- Relies on the assumption that the black-box VLM is a CLIP-style model with a residual/shortcut structure for the collaborative decomposition to hold.
- Evaluated only on few-shot image classification; results on other tasks are not reported.
- Still trails the white-box CoOp on average (about 1.62% on ImageNet, and CoOp is better in the distribution-shift table on several sets).
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft EuroSAT UCF101 DTD SUN397 ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft EuroSAT UCF101 DTD SUN397 ImageNetV2
Metrics: accuracy training time memory footprint
Baselines: ZS-CLIP BBT CoOp BlackVIP CBBT
Builds on: Black-Box Tuning (BBT) CoOp CLIP-Adapter CMA-ES
Closest papers by meaning
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.93)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.91)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.90)
- Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves (CVPR 2025 · similarity 0.90)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.