Research map / Vision-language and multimodal / CLIP and zero-shot
ProAPO: Progressively Automatic Prompt Optimization for Visual Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProAPO, an evolution-based automatic prompt optimization that progressively refines CLIP prompts from task-level templates to class-specific descriptions using one-shot supervision, with edit/evolution operators, an entropy-constrained fitness score, and two sampling strategies.
Problem
Class-specific prompts for VLM image classification are often inaccurate or non-discriminative due to LLM hallucination, and optimizing them suffers from high generation cost, long iteration times, and overfitting under minimal (one-shot) supervision.
Approach
Progressively optimize prompts from templates to class-specific descriptions via edit-based (add/delete/replace) and evolution-based (crossover/mutation) operators over a prompt library generated by a one-time LLM query, scored by a fitness function combining accuracy with an entropy term, plus prompt and group sampling to reduce cost.
Key results
- ProAPO reaches 64.6% average accuracy over 11 datasets and 61.8% over 13 with ResNet-50 (vs. 53.4% vanilla CLIP), and 67.9% / 65.0% with ViT-B/32.
- With ViT-B/32, ProAPO improves CuPL by 2.7% average across 13 datasets and DCLIP by 3.3%.
- Sampling strategies: removing group sampling raises time from 15 to 306 minutes with similar accuracy; removing prompt sampling lowers accuracy (avg 13 from 65.0 to 64.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| optimize prompts progressively from templates to class-specific descriptions | CLIP ViT-B/32 · 13 image-classification datasets | average top-1 accuracy (%): 59.3 -> 65.0 | Helped |
| optimize prompts progressively from templates to class-specific descriptions | CLIP ResNet-50 · 13 image-classification datasets | average top-1 accuracy (%): 53.4% to 61.8% | Helped |
| add class-specific prompt optimization to DCLIP initialization | DCLIP · 13 image-classification datasets | average top-1 accuracy (%): +3.3% | Helped |
| add class-specific prompt optimization to CuPL initialization | CuPL · 13 image-classification datasets | average top-1 accuracy (%): +2.7% | Helped |
| add class-specific prompt optimization to DCLIP initialization | DCLIP · EuroSAT | top-1 accuracy (%): +21% | Helped |
| add crossover to add/delete/replace prompt generation (ablation) | ProAPO · Avg (13) | top-1 accuracy (%): 64.0 -> 64.5 | Helped |
| add mutation to add/delete/replace/crossover prompt generation (ablation) | ProAPO · Avg (13) | top-1 accuracy (%): 64.5 -> 65.0 | Helped |
| add delete and replace operations to add-only generation (ablation) | ProAPO · Avg (13) | top-1 accuracy (%): 63.3 -> 64.0 | Helped |
| remove prompt sampling (ablation) | ProAPO · Avg (13) | top-1 accuracy (%): 65.0 -> 64.5 | Hurt |
| remove prompt sampling (ablation) | ProAPO · ImageNet-1K | top-1 accuracy (%): 64.7 -> 64.4 | Hurt |
| remove group sampling (ablation) | ProAPO · Avg (13) | top-1 accuracy (%): 65.0 -> 65.2 | Helped |
| remove group sampling (ablation) | ProAPO · optimization time | time: 15 min -> 306 min | Hurt |
Surprising findings
- Ensembled templates have a higher upper bound than a single template, and the optimized template even beats the best single hand-crafted template.
- CoOp prompts transfer poorly across backbones (e.g., ImageNet accuracy drops to 20.6% on RN101), while searched natural-language prompts transfer well.
Limitations
- Gains are small on coarse-grained datasets (ImageNet, Caltech) and scene datasets (Places, SUN), attributed to limits of text-only optimization.
- Relies on one-shot supervision; prompt quality evaluation may still overfit to the few training samples.
- Optimization is limited to the textual side and the fixed language search space (CuPL/Template-80 libraries).
Experimental setup
Datasets: ImageNet-1K Caltech101 StanfordCars CUB200-2011 DTD EuroSAT FGVC-Aircraft Flowers102 Food101 Oxford-Pets Places365 SUN397
Benchmarks: ImageNet-1K Caltech101 StanfordCars CUB200-2011 DTD EuroSAT FGVC-Aircraft Flowers102 Food101 Oxford-Pets Places365 SUN397
Metrics: top-1 accuracy average accuracy Pearson correlation coefficient
Baselines: CLIP CoOp PLOT ProGrad PN Template-80 FILIP-8 DEFILIP-6 DCLIP Waffle CuPL GPT4Vis
Builds on: PN CuPL CLIP DCLIP Template-80
Closest papers by meaning
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.91)
- Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation (ICLR 2025 · similarity 0.91)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.91)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.90)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.