Research map / Image generation / Text-to-image diffusion
Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Visually Guided Decoding (VGD), a gradient-free hard prompt inversion method that uses LLM token generation guided by CLIP similarity to produce interpretable Stable Diffusion prompts.
Problem
Hard and soft prompt inversion for text-to-image models yields non-interpretable or incoherent prompts and requires costly gradient-based optimization.
Approach
Decode prompts token-by-token with beam search where the score combines CLIP image-text similarity (approximating P(I|T)) with LLM next-token probability (prior P(T)^alpha), so no prompt embedding is trained.
Key results
- On LAION-400M, VGD with LLaVA-1.5 reaches CLIP-I 0.573 vs 0.538 for PEZ (16 tokens) and 0.540 for LLaVA+CLIP Interrogator, achieving SOTA CLIP-I across all four datasets.
- VGD (LLaMA2-7B, 16 tokens) reaches MS COCO BERTScore F1 0.847 versus PEZ's 0.795 at 16 tokens.
- VGD generates a prompt in 15.33 s versus 193.14 s for PEZ and 34264.74 s for PH2P on a single A100 80GB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VGD instead of the strongest listed baseline | Stable Diffusion 2.1-768 prompt inversion · LAION-400M | CLIP-I score: 0.538 -> 0.511 | Hurt |
| Use VGD instead of the strongest listed baseline | Stable Diffusion 2.1-768 prompt inversion · MS COCO | CLIP-I score: 0.685 -> 0.678 | No clear effect |
| Use VGD instead of the strongest listed baseline | Stable Diffusion 2.1-768 prompt inversion · Celeb-A | CLIP-I score: 0.622 -> 0.514 | Hurt |
| Use VGD instead of the strongest listed baseline | Stable Diffusion 2.1-768 prompt inversion · Lexica.art | CLIP-I score: 0.762 -> 0.753 | No clear effect |
| Use VGD instead of PEZ | Prompt inversion · MS COCO | BERTScore F1: 0.795 -> 0.847 | Helped |
| Use VGD instead of PEZ | Prompt inversion · Lexica.art | BERTScore F1: 0.777 -> 0.802 | Helped |
| Use VGD instead of PEZ | Prompt generation · Prompt generation | time (sec): 193.14 -> 15.33 | Helped |
| Remove CLIP guidance (ablation) | VGD · Lexica.art | CLIP score: 0.735 -> 0.487 | Hurt |
| Remove CLIP guidance (ablation) | VGD · Lexica.art | BERTScore F1: 0.801 -> 0.791 | No clear effect |
| Remove LLM language-model guidance (ablation) | VGD · Lexica.art | CLIP score: 0.735 -> 0.768 | Helped |
| Remove LLM language-model guidance (ablation) | VGD · Lexica.art | BERTScore F1: 0.801 -> 0.751 | Hurt |
| Replace the aligned CLIP encoder with openai/ViT-B-32 (ablation) | VGD · Lexica.art | CLIP score: 0.735 -> 0.655 | Hurt |
Surprising findings
- CLIP-only decoding achieves the highest CLIP score (0.768) but the lowest BERTScore F1 (0.751), showing that CLIP-only decoding yields less coherent prompts.
- Swapping in a different CLIP model for guidance (ViT-B-32 or ViT-L-14) substantially lowers CLIP-I scores (0.655 and 0.626 vs 0.735), indicating the approximation depends on the CLIP encoder used by Stable Diffusion.
Limitations
- Evaluation focuses on Stable Diffusion and Midjourney; generalization to other T2I models is only partly shown.
- Struggles to capture fine-grained details of regional or background objects in complex images with multiple objects.
- Dependence on the specific CLIP encoder: misaligned CLIP models degrade performance.
Experimental setup
Datasets: LAION-400M MS COCO Celeb-A Lexica.art
Benchmarks: LAION-400M MS COCO Celeb-A Lexica.art
Metrics: CLIP-I score BERTScore perplexity human preference
Baselines: PEZ Textual Inversion PH2P LLaVA-1.5 LLaVA-1.5 + CLIP Interrogator
Builds on: PEZ Textual Inversion noisy channel model CLIP
Closest papers by meaning
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.91)
- PromptCoT: Align Prompt Distribution via Adapted Chain-of-Thought (CVPR 2024 · similarity 0.90)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Imagine Before You Draw: Visual Prompt Engineering for Image Generation (NeurIPS 2026 · similarity 0.89)
- A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis (AAAI 2024 · similarity 0.88)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.