Research map / Vision-language and multimodal / CLIP and zero-shot

DPC: Dual-Prompt Collaboration for Tuning Vision-Language Models

CVPR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Prompt learning for CLIP · 15 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Dual-Prompt Collaboration (DPC), a plug-and-play method that clones a parallel prompt from a CLIP prompt-tuning backbone and decouples base and new class optimization via weighted mixing plus a hard-negative contrastive optimizer.

Problem

The Base-New Trade-off in CLIP prompt tuning: fine-tuning on base classes degrades generalization to unseen new classes because base and new optimization directions interfere when they share the same prompt.

Approach

Decouple base and new optimization at the prompt level by cloning a learnable parallel prompt from the tuned backbone prompt, optimizing it with a dynamic hard-negative symmetric contrastive loss on base classes, and mixing the frozen tuned prompt and parallel prompt with separate weights (omega_b for base, omega_n for new) at inference.

Key results

• On base-to-new generalization over 11 datasets, DPC raises CoOp's average base accuracy from 81.98 to 85.15 while keeping new accuracy at 68.84 (HM 74.84 to 76.13).
• Applied to PromptKD, DPC improves average base accuracy from 86.86 to 87.55 with new accuracy unchanged at 80.55.
• DPC yields a higher harmonic-mean gain than DePT across the three baselines in the plug-and-play comparison (Fig. 5).

Measured effects reported in the paper

Surprising findings

• Applying DPC to PromptKD slightly reduces base accuracy on some datasets (e.g., Flowers102, DTD-like drops such as SUN397 and UCF101 HM-level changes are mixed), showing gains are not uniform across backbones.

Limitations

• Mixing weights omega_b and omega_n are hand-tuned hyperparameters; adaptive weighting is left to future work.
• Gains are not uniform across backbones and datasets (e.g., PromptKD shows small decreases on some datasets).

Experimental setup

Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101

Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNet-V2

Metrics: accuracy harmonic mean

Baselines: CoOp MaPLe PromptSRC PromptKD DePT

Builds on: CoOp CoCoOp MaPLe PromptSRC PromptKD

Closest papers by meaning

• Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.91)
• IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.90)
• Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.90)
• OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning (ICML 2025 · similarity 0.89)
• ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.89)
• Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves (CVPR 2025 · similarity 0.89)