Research map / Image generation / Text-to-image diffusion
Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PEZ hard prompts rather than CLIP Interrogator prompts | Stable Diffusion-v2 image generation · Celeb-A | CLIP score: $0.558$ -> $0.602$ | Helped |
| Use PEZ hard prompts rather than CLIP Interrogator prompts | Stable Diffusion-v2 image generation · LAION, MS COCO, and Lexica.art | CLIP score: LAION $0.707$ vs $0.697$; MS COCO $0.690$ vs $0.674$; Lexica.art $0.762$ vs $0.711$ | Mixed |
| Maintain continuous prompt iterates and project only for the forward pass | PEZ · LAION | CLIP score: $0.697$ vs AutoPrompt SGD $0.689$ and FluentPrompt $0.688$ | Helped |
| Add a keyword-bank constraint to PEZ (ablation) | PEZ image prompts · Celeb-A | CLIP score: $0.602$ -> $0.629$ | Helped |
| Add a keyword-bank constraint to PEZ (ablation) | PEZ image prompts · LAION, MS COCO, and Lexica.art | CLIP score: LAION $0.697$ -> $0.702$; MS COCO $0.674$ -> $0.689$; Lexica.art $0.711$ -> $0.740$ | Helped |
| Reduce CLIP Interrogator prompt length from approximately 77 tokens to 8 (ablation) | CLIP Interrogator · LAION, MS COCO, Celeb-A, and Lexica.art | CLIP score: LAION $0.707$ -> $0.539$; MS COCO $0.690$ -> $0.575$; Celeb-A $0.558$ -> $0.360$; Lexica.art $0.762$ -> $0.532$ | Hurt |
| Replace the hard prompt with an 8-token soft prompt (ablation) | Image prompt optimization · LAION, MS COCO, Celeb-A, and Lexica.art | CLIP score: LAION $0.697$ -> $0.408$; MS COCO $0.674$ -> $0.420$; Celeb-A $0.602$ -> $0.451$; Lexica.art $0.711$ -> $0.554$ | Hurt |
| Increase CLIP Interrogator prompt length from 8 tokens to 16 or 32 (ablation) | CLIP Interrogator · LAION, MS COCO, Celeb-A, and Lexica.art | CLIP score: 8 tokens: $0.539$, $0.575$, $0.360$, $0.532$; 16 tokens: $0.650$, $0.650$, $0.491$, $0.671$; 32 tokens: $0.694$, $0.663$ | Helped |
| Add a fluency constraint to PEZ (ablation) | SST-2 prompts transferred from GPT-2 Large · SST-2 validation on GPT-2 XL | accuracy: $77.8_{\pm3.45}$ -> $\textbf{79.72}_{\pm3.26}$ | Helped |
| Add a fluency constraint to PEZ (ablation) | SST-2 prompts trained on GPT-2 Large · SST-2 validation on GPT-2 Large | accuracy: $88.12_{\pm0.15}$ -> $88.05_{\pm0.55}$ | No clear effect |
| Add a fluency constraint to PEZ (ablation) | few-shot AGNEWS prompts · AGNEWS validation, k=2 and k=4 | accuracy: $70.07_{\pm0.81}$ -> $70.93_{\pm0.60}$; $73.99_{\pm0.45}$ -> $74.15_{\pm0.48}$ | Helped |
| Add a fluency constraint to PEZ (ablation) | 10-token AGNEWS prompt · AGNEWS validation | accuracy: $77.06_{\pm 0.20}$ -> $76.94_{\pm 0.48}$ | Hurt |
Closest papers by meaning
- A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis (AAAI 2024 · similarity 0.91)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- On Discrete Prompt Optimization for Diffusion Models (ICML 2024 · similarity 0.90)
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
- The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation (CVPR 2025 · similarity 0.89)
- Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.