Research map / Vision-language and multimodal / CLIP and zero-shot
Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Surrogate Prompt Learning (SurPL), which learns one basic prompt and uses a lightweight cross-attention Surrogate Feature Generator to produce diverse prompted text features from conditional signals, avoiding per-instance/per-region text-encoder backpropagation.
Problem
Diverse prompt learning for vision-language models (instance-dependent and fine-grained prompts) improves transfer but demands large GPU memory and training time because gradients must flow through the text encoder for many prompted text features.
Approach
Instead of learning diverse text prompts, a Surrogate Feature Generator (a cross-attention module over the basic prompted text feature, with conditional signals as keys/values) directly generates diverse text features: instance-dependent signals from visual features and learnable fine-grained signals, keeping text-encoder backprop cost near single-prompt methods.
Key results
- SurPL achieves 85.12% average few-shot accuracy over 11 datasets, versus 84.40% for GalLoP and 82.92% for the DVLP baseline.
- SurPL-G reaches 81.03% harmonic mean on base-to-novel generalization, improving 0.61% over DePT (80.42%).
- On EuroSAT, SurPL trains with 5.60G GPU memory versus 4.69G for GalLoP and 13.02G for PLOT++ at similar batch size, and runs ImageNet on a single 24G RTX 3090 where diverse methods run out of memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SurPL instead of GalLoP | CLIP ViT-B/16 prompt learning · 11-dataset few-shot average | accuracy: 84.40 -> 85.12 | Helped |
| Use SurPL instead of GalLoP | CLIP ViT-B/16 prompt learning · FGVCAircraft | accuracy: 58.30 -> 60.51 | Helped |
| Use SurPL instead of GalLoP | CLIP ViT-B/16 prompt learning · ImageNet | accuracy: 75.10 -> 74.65 | Hurt |
| Use SurPL-G instead of DePT | CLIP ViT-B/16 base-to-novel generalization · 11-dataset average | harmonic mean: 80.42 -> 81.03 | Helped |
| Use SurPL-G instead of MaPLe | CLIP ViT-B/16 cross-dataset generalization · Target average | accuracy: 66.30 -> 66.61 | Helped |
| Use SurPL-G instead of PSRC | CLIP ViT-B/16 cross-domain generalization · Target average | accuracy: 60.65 -> 61.37 | Helped |
| Add global-invariant text feature (ablation) | SurPL over DVLP · Few-shot, 11-dataset average | accuracy: 82.92 -> 83.70 | Helped |
| Add instance-dependent feature to global-invariant feature (ablation) | SurPL · Few-shot, 11-dataset average | accuracy: 83.70 -> 83.96 | Helped |
| Add fine-grained feature to global-invariant feature (ablation) | SurPL · Few-shot, 11-dataset average | accuracy: 83.70 -> 84.99 | Helped |
| Add instance-dependent feature to global-invariant feature (ablation) | SurPL-G · Base classes, 11-dataset average | accuracy: 81.84 -> 84.77 | Helped |
| Add fine-grained feature to global-invariant feature (ablation) | SurPL-G · Base classes, 11-dataset average | accuracy: 81.84 -> 85.52 | Helped |
| Add instance-dependent feature to global-invariant feature (ablation) | SurPL-G · Novel classes, 11-dataset average | accuracy: 74.68 -> 76.19 | Helped |
Surprising findings
- Efficiency gains over diverse prompt learning methods come with accuracy gains, not just reduced cost; e.g., CoCoOp runs out of memory on ImageNet and Aircraft in the comparison.
Limitations
- The Surrogate Feature Generator is a cross-attention module, which is relatively outdated and increases parameter count.
- The method incorporates existing diverse prompt learning frameworks (instance-dependent and fine-grained) rather than discovering new perspectives of vision-language alignment.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNetV2
Metrics: top-1 accuracy harmonic mean GPU memory training time testing time
Baselines: CoOp CoCoOp MaPLe DAPT PSRC LLaMP PLOT++ ALIGN GalLoP ProGrad KgCoOp DePT
Builds on: CoOp CoCoOp DVLP GalLoP Sharpness-aware Minimization
Closest papers by meaning
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.91)
- Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior (AAAI 2024 · similarity 0.91)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- ProAPO: Progressively Automatic Prompt Optimization for Visual Classification (CVPR 2025 · similarity 0.89)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.89)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.