Research map / Large language models / Language, safety and interpretability

Language Models as Black-Box Optimizers for Vision-Language Models

CVPR 2024 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 22 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses chat-based LLMs (ChatGPT) as black-box optimizers that iteratively refine natural-language class-agnostic prompts for CLIP via hill-climbing with positive and negative conversational feedback, and extends to DALL-E 3 text-to-image optimization with GPT-4V feedback.

Problem

Optimizing vision-language models (e.g., CLIP, DALL-E 3) when weights, embeddings, and logits are inaccessible, so white-box fine-tuning methods cannot be used; only natural-language prompts can be tuned for few-shot classification and text-to-image generation.

Approach

Treat a chat LLM as the optimizer in a stochastic hill-climbing loop: it sees top-k and bottom-k scored prompts with their accuracies and proposes improved templates, using the conversational feedback as an implicit gradient direction, with random restarts and resets for exploration.

Key results

• On 1-shot classification across 11 datasets, Ours (P+N) averages 61.1% vs CoOp 59.6% and WiSE-FT 59.1%, and improves initial LAION-COCO prompts from 56.0% to 61.1%.
• Black-box prompts transfer from RN50 to ViT-B/16 at 67.0% on 16-shot ImageNet, versus CoOp's continuous prompts dropping to 39.5%.
• Human evaluation with DALL-E 3 after three optimization rounds raises T2I faithfulness from 2.62 to 3.56 and prompt inversion from 1.94 to 3.68 on a 1-5 scale.

Measured effects reported in the paper

Surprising findings

• Including negative (lowest-scoring) prompts alongside positive ones improves efficiency and accuracy, suggesting LLMs use textual feedback as an implicit gradient direction.
• CoOp's continuous prompts lose large accuracy when transferred across CLIP backbones (e.g., RN101 falls to 20.6%), while natural-language prompts transfer well.

Limitations

• Energy and API cost of LLM queries remains a substantial concern.
• Human evaluation for T2I is costly and small-scale; no automatic evaluation.
• Lower than the SOTA white-box Cross-Modal Adaptation method; GPT4-V and DALL-E 3 fail on abstract details and object counts.

Experimental setup

Datasets: ImageNet Caltech101 FGVC-Aircraft Food101 Oxford Pets Stanford Cars SUN397 UCF101 DTD EuroSAT Flowers102 LAION-COCO

Benchmarks: CoOp 11-dataset few-shot classification (1-shot) ImageNet 16-shot prompt transfer Winoground-based text-to-image DiffusionDB prompt inversion

Metrics: accuracy human faithfulness score (1-5 Likert)

Baselines: CoOp WiSE-FT Cross-Modal Adaptation LFA BlackVIP DCLIP Hand-Engineered prompts a photo of a {}

Builds on: CoOp Algorithm 1 human prompt engineering APE DCLIP Cross-Modal Adaptation

Closest papers by meaning

• Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models (ICML 2024 · similarity 0.93)
• IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.93)
• Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.91)
• ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.91)
• ProAPO: Progressively Automatic Prompt Optimization for Visual Classification (CVPR 2025 · similarity 0.91)
• Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.91)