Research map / Vision-language and multimodal / CLIP and zero-shot
Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AAPE, a prompt-learning method that distills LLM- or human-written natural language prompts into an image-conditioned prompt embedding via a learned attention aggregator and a prompt generator, removing LLM cost at test time.
Problem
CLIP adapts poorly to specialized domains and fine-grained classes under few-shot and OOD settings; existing LLM-augmented methods incur LLM inference cost at test time and use noisy, task-irrelevant text.
Approach
Learn an attention-based, image-aligned aggregation of reference prompts (CLIP-reward supervised), then train a lightweight prompt generator conditioned on the image to match the aggregated summary (distillation) while minimizing task loss; discard the LLM and aggregator at test time.
Key results
- Base-to-new few-shot classification averaged over 11 datasets: H = 80.97 vs. 80.34 for OGEN, a new SOTA in the paper's comparison.
- Zero-shot Flickr30k text retrieval R@1 of 90.8 for AAPE, competitive with CLIP (88.0) and SigLIP (91.5).
- LiMBeR+AAPE zero-shot on NoCaps CIDEr-D overall 45.1 vs. 43.9 for LiMBeR; train/finetune variant reaches 47.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill LLM-generated natural-language prompts and adapt them with task loss (ablation) | CLIP prompt learning · Base-to-new, 11-dataset average | Harmonic mean: 76.23±0.22 -> 80.97±0.19 | Helped |
| add the class-template embedding and projection g to h(x)-only (ablation) | AAPE classification · Base-to-new, 11-dataset average | Harmonic mean: 79.77±0.13 -> 80.97±0.19 | Helped |
| use attention-based rather than MLP-based prompt aggregation (ablation) | AAPE prompt aggregator · Base-to-new, 11-dataset average | Harmonic mean: 80.16±0.22 -> 80.97±0.19 | Helped |
| use attention-based rather than mean prompt aggregation (ablation) | AAPE prompt aggregator · Base-to-new, 11-dataset average | Harmonic mean: 78.71±0.15 -> 80.97±0.19 | Helped |
| use attention-based rather than randomly sample a reference prompt (ablation) | AAPE prompt aggregator · Base-to-new, 11-dataset average | Harmonic mean: 72.30±1.15 -> 80.97±0.19 | Helped |
| increase distillation weight from 1 to values in [3,9] (ablation) | AAPE · Base-to-new, 11-dataset average | Harmonic mean: 79.14±0.27 -> 80.65±0.23 | Helped |
| use learned aggregate-and-adapt prompts instead of learning-free CuPL prompt averaging | CLIP few-shot classification · Base-to-new, 11-dataset average | Harmonic mean: 74.78 -> 80.97±0.19 | Helped |
| add AAPE to LiMBeR | Image captioning on COCO · COCO | CIDEr-D: 54.9 -> 57.8 | Helped |
| add AAPE to LiMBeR | VQA2 · VQA2, 0-shot | accuracy: 33.3 -> 36.5 | Helped |
| transfer COCO-trained AAPE zero-shot to NoCaps | LiMBeR image captioning · NoCaps, All | CIDEr-D: 43.9 -> 45.1 | Helped |
Surprising findings
- Modality gap is not highly correlated with downstream generalization: AAPE improves accuracy while optimal gap changes in inconsistent directions across datasets.
- Random sampling of reference prompts yields only marginal gains over zero-shot CLIP, showing noisy LLM prompts are harmful without aggregation.
Limitations
- Aggregating into a single prompt embedding (m=1) may not model text diversity when many prompts are available; multiple embeddings are harder to learn on data-deficient tasks.
- Input-conditional prompt prediction makes classification inference slow compared with fixed prompt learners.
- Study limited to CLIP; applicability to other vision-language models is untested.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVC-Aircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVC-Aircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Metrics: accuracy harmonic mean Recall@K CIDEr-D CLIP-S Ref-CLIPScore VQA accuracy FPS GFLOP
Baselines: CoOp CoCoOp MaPLe CLIPood PromptSRC OGEN ProGrad KgCoOp LASP-V CuPL ProText ArGue-N
Builds on: CoCoOp CuPL CoOp LiMBeR
Closest papers by meaning
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.93)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- Concept-Guided Prompt Learning for Generalization in Vision-Language Models (AAAI 2024 · similarity 0.91)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.90)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · similarity 0.90)
- Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.