Research map / Image generation / Text-to-image diffusion
On Discrete Prompt Optimization for Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DPO-Diff, a gradient-based discrete prompt optimizer for text-to-image diffusion models that uses compact word-substitution spaces, a constant-cost Shortcut Text Gradient, and evolutionary search, for both prompt enhancement and adversarial attack.
Problem
Prompt engineering for text-to-image diffusion models is manual and trial-and-error, and exact text gradients through the diffusion sampling chain and discrete embedding lookup are prohibitively expensive and non-differentiable.
Approach
Formulate prompt optimization as discrete search over compact, dynamically generated synonym/antonym word spaces; approximate the text gradient with a K-step shortcut that estimates x0 from an intermediate latent (constant memory and runtime), relax word choice with Gumbel-Softmax, and refine candidates with evolutionary search. Optimizing negative prompts via antonyms is found more effective than
Key results
- Prompt improvement: DPO-Diff reaches HPSv2 62.37 on DiffusionDB vs 54.39 for Promptist, with lower CLIP loss (0.81 vs 0.88).
- Human evaluation: DPO-Diff surpasses or matches Promptist 79% of the time on SD-v1 and 88% on SD-XL.
- Adversarial attack achieves a 44% human-judged success rate on SD-v1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize negative prompts with Antonyms Space rather than positive prompts with Synonym Space (ablation) | DPO-Diff · DiffusionDB | CLIP loss: 0.8747 ± 0.0189 -> 0.8579 ± 0.0242 | Helped |
| Optimize negative prompts with Antonyms Space rather than positive prompts with Synonym Space (ablation) | DPO-Diff · ChatGPT | CLIP loss: 0.8304 ± 0.0284 -> 0.8133 ± 0.0197 | Helped |
| Optimize negative prompts with Antonyms Space rather than positive prompts with Synonym Space (ablation) | DPO-Diff · COCO | CLIP loss: 0.8624 ± 0.0141 -> 0.8403 ± 0.0210 | Helped |
| Use DPO-Diff prompt improvement instead of Promptist | Stable Diffusion v1-4 · DiffusionDB | CLIP loss: 0.88 ± 0.02 -> 0.81 ± 0.03 | Helped |
| Use DPO-Diff prompt improvement instead of Promptist | Stable Diffusion v1-4 · COCO | HPSv2: 50.08 ± 7.43 -> 61.26 ± 0.77 | Helped |
| Use DPO-Diff prompt improvement instead of Promptist | Stable Diffusion v1-4 · ChatGPT | HPSv2: 59.32 ± 6.50 -> 67.71 ± 6.46 | Helped |
| Reverse the objective to optimize adversarial prompts | DPO-Diff · DiffusionDB adversarial attack | CLIP: 0.76 ± 0.03 -> 0.86 ± 0.05 | Helped |
| Reverse the objective to optimize adversarial prompts | DPO-Diff · COCO adversarial attack | CLIP: 0.77 ± 0.03 -> 0.94 ± 0.04 | Helped |
| Reverse the objective to optimize adversarial prompts | DPO-Diff · ChatGPT adversarial attack | CLIP: 0.77 ± 0.02 -> 0.95 ± 0.05 | Helped |
| Combine GPO with evolutionary search (EPO) rather than using either search method alone (ablation) | DPO-Diff · Prompt optimization search budgets | performance | Helped |
| Use Synonym Space for positive prompt optimization instead of Antonyms Space for negative prompt optimization (ablation) | DPO-Diff · Prompt improvement across DiffusionDB, ChatGPT, and COCO | CLIP loss: 0.8579 ± 0.0242 vs 0.8747 ± 0.0189; 0.8133 ± 0.0197 vs 0.8304 ± 0.0284; 0.8403 ± 0.0210 vs 0.8624 ± 0.0141 | Hurt |
| Use human-engineered prompts instead of the original user prompt | Stable Diffusion v1-4 · DiffusionDB prompt improvement | CLIP loss: 0.87 ± 0.02 -> 0.89 ± 0.04 | Hurt |
Surprising findings
- Optimizing negative prompts (antonym space) yields lower CLIP loss than optimizing positive prompts (synonym space).
- Human-engineered and Promptist-optimized prompts do not improve prompt-image relevance, mostly adding aesthetic modifiers.
Limitations
- Search requires multiple diffusion passes per prompt, so inference is slow compared to learned prompt rewriters.
- Performance is bounded by the CLIP text encoder, which discards spatial relations.
- CLIP loss may not align with human judgment; ChatGPT synonyms can be incorrect for adversarial attacks.
Experimental setup
Datasets: DiffusionDB COCO 2014 validation captions ChatGPT-generated prompts
Benchmarks: DiffusionDB prompts COCO prompts ChatGPT prompts
Metrics: spherical CLIP loss HPSv2 human win rate attack success rate
Baselines: Promptist Manual (human-engineered prompts) Random Search Evolution Prompt Optimization
Builds on: classifier-free guidance ReFL Gumbel-Softmax Promptist CLIP
Closest papers by meaning
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.93)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.93)
- Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.91)
- Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.91)
- Minority-Focused Text-to-Image Generation via Prompt Optimization (CVPR 2025 · similarity 0.91)
- Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.