Research map / Vision-language and multimodal / CLIP and zero-shot
IPO: Interpretable Prompt Optimization for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IPO, which uses an LLM (GPT-3.5 Turbo) as a parameter-free prompt optimizer with episodic memory of past prompts and LMM-generated image descriptions (MiniCPM-V-2.0) to produce interpretable CLIP prompts.
Problem
Gradient-based prompt learning for CLIP overfits base classes and yields prompts (learned token vectors) that humans cannot interpret.
Approach
Replace gradient descent on continuous prompt vectors with an LLM that iteratively proposes natural-language prompts, conditioned on top-scoring past prompts with their accuracy/loss (episodic memory) and on LMM-generated descriptions of base-class training images.
Key results
- 1-shot IPO achieves 74.29 H averaged over 11 datasets vs. 71.70 for CLIP, with novel accuracy 77.00 (best among compared 1-shot methods).
- 16-shot IPO achieves 80.51 novel accuracy and 80.21 H, versus 79.97 H for PromptSRC and 80.48 H for CoPrompt (CoPrompt slightly higher in H per table 'Comparison with gradient-based' where CoPrompt H=80.48 exceeds IPO's 80.21).
- Replacing GPT-3.5 Turbo with GPT-4o in the appendix raises the 1-shot average H to 75.42-76.06 range (e.g., 76.06 with GPT-4o LLM and GPT-4o LMM).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IPO instead of CLIP | CLIP · 11 datasets, 1-shot average | H: 71.70 -> 74.29 | Helped |
| Use IPO instead of CoPrompt | CLIP ViT-B/16 · 11 datasets, 16-shot average | Novel accuracy: 77.23 -> 80.51 | Helped |
| Add LMM-generated image descriptions (ablation) | IPO · 11 datasets, 16-shot average | H: 73.49 -> 74.29 | Helped |
| Add prompt history rather than use no history (ablation) | IPO · 11 datasets, 16-shot average | H: 72.04 -> 74.29 | Helped |
| Increase history length from n=20 to n=100 (ablation) | IPO · 11 datasets, 16-shot average | H: 74.29 -> 74.33 | Helped |
| Replace GPT-3.5-turbo and MiniCPM-V-2 with GPT-4o components (ablation) | IPO · 11 datasets, average | H: 74.29 -> 76.06 | Helped |
| Replace GPT-3.5 Turbo with GPT-4o (ablation) | IPO · ImageNet, large-scale generic dataset | H: 71.54 -> 74.09 | Helped |
| Use IPO with GPT-3.5 Turbo rather than the stronger comparison methods | ImageNet · ImageNet, 1-shot | H: 72.39 -> 71.54 | Hurt |
| Replace GPT-3.5-turbo optimizer with GPT-4o (ablation) | IPO · 11 datasets, average | Novel accuracy: 77.00 -> 78.13 | Helped |
| Remove learned CoOp prompt tokens (ablation) | CoOp · Flower102 novel classes | Novel accuracy: 72.47 -> 75.83 | Helped |
| Add the word “Identify” to the prompt (ablation) | IPO prompt on Flower102 · Flower102 novel classes | Novel accuracy: 75.83 -> 77.52 | Helped |
| Use “a photo <CLASS>.” instead of “a photo of a <CLASS>.” (ablation) | CLIP prompt · Flower102 novel classes | Novel accuracy: 76.72 -> 77.25 | Helped |
Surprising findings
- Removing learned tokens from CoOp and CoCoOp improves novel-class accuracy, indicating base-class overfitting.
- All six LMMs tried for direct prompt optimization failed to follow the meta-prompt and instead produced image descriptions.
Limitations
- Scaling to large datasets requires very long LLM inputs (about 5,000 to 50,000 tokens), raising API cost.
- Primarily designed for few-shot scenarios; image descriptions omitted in 16-shot setting due to input length limits.
- Relies on closed-source GPT-3.5 Turbo for main results, with cost and latency dependence on API.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft EuroSAT UCF101 DTD SUN397
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft EuroSAT UCF101 DTD SUN397
Metrics: base accuracy novel accuracy harmonic mean (H)
Baselines: CLIP CoOp CoCoOp MaPLe PromptSRC CoPrompt VPT + L2P LFA PLOT
Builds on: OPRO CoOp CoCoOp CLIP
Closest papers by meaning
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.93)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.93)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.92)
- Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP (NeurIPS 2024 · similarity 0.92)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.92)
- ProAPO: Progressively Automatic Prompt Optimization for Visual Classification (CVPR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.