Research map / Image generation / Text-to-image diffusion
Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Inverts a frozen latent diffusion model (Stable Diffusion v1.5) into interpretable hard text prompts by delayed-projection L-BFGS optimization restricted to later timesteps (t>=500).
Problem
Recovering interpretable text prompts that would generate a target image, since embedding-based textual inversion yields pseudo-tokens lacking semantics and readability; discrete prompt optimization through diffusion models is hard due to vanishing gradients and high-variance trajectories.
Approach
Optimize discrete vocabulary tokens directly using the diffusion denoising loss (no auxiliary image-text model), restricting the timestep range to later noisy steps (t>=500) where semantics are encoded, and using projected gradient with delayed projection and L-BFGS updates.
Key results
- PH2P achieves CLIP similarity 0.77 versus 0.72 for PEZ on COCO and 0.77 versus 0.70 on LSUN/SUN.
- PH2P BERTScore F1 of 0.820 on COCO versus 0.802 for PEZ, with precision 0.803 versus 0.772.
- Adam-based variant reaches only 0.65 CLIP similarity, and optimizing all timesteps reaches 0.72, both below PH2P's 0.77.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PH2P hard-prompt inversion instead of PEZ | Stable Diffusion v1.5 · COCO | CLIP similarity: 0.72 -> 0.77 | Helped |
| Use PH2P hard-prompt inversion instead of PEZ | Stable Diffusion v1.5 · COCO | LPIPS similarity: 0.477 -> 0.462 | Helped |
| Use PH2P hard-prompt inversion instead of PEZ | Stable Diffusion v1.5 · COCO | LPIPS diversity: 0.417 -> 0.435 | Helped |
| Use PH2P hard-prompt inversion instead of PEZ | Stable Diffusion v1.5 · LSUN | CLIP similarity: 0.70 -> 0.77 | Helped |
| Use PH2P hard-prompt inversion instead of PEZ | Stable Diffusion v1.5 · LSUN | LPIPS similarity: 0.480 -> 0.463 | Helped |
| Use PH2P hard-prompt inversion instead of PEZ | Stable Diffusion v1.5 · LSUN | LPIPS diversity: 0.420 -> 0.422 | Helped |
| Replace the PH2P optimizer with Adam (ablation) | PH2P prompt inversion · COCO | CLIP similarity: 0.77 -> 0.65 | Hurt |
| Replace the PH2P optimizer with Adam (ablation) | PH2P prompt inversion · COCO | LPIPS similarity: 0.462 -> 0.501 | Hurt |
| Replace the PH2P optimizer with Adam (ablation) | PH2P prompt inversion · COCO | LPIPS diversity: 0.435 -> 0.400 | Hurt |
| Optimize over all diffusion timesteps instead of the selected timestep range (ablation) | PH2P prompt inversion · COCO | CLIP similarity: 0.77 -> 0.72 | Hurt |
| Optimize over all diffusion timesteps instead of the selected timestep range (ablation) | PH2P prompt inversion · COCO | LPIPS similarity: 0.462 -> 0.479 | Hurt |
| Optimize over all diffusion timesteps instead of the selected timestep range (ablation) | PH2P prompt inversion · COCO | LPIPS diversity: 0.435 -> 0.423 | Hurt |
Surprising findings
- Optimizing over all diffusion timesteps, including early ones, adds no new semantic information and tends to produce special characters in prompts.
- Standard Adam with projection performs notably worse than L-BFGS with delayed projection for hard prompt inversion.
Limitations
- Evaluation is limited to 500 validation images from COCO and SUN across 5 seeds.
- Relies on PEZ as the only direct hard-prompt baseline, which is a concurrent unpublished work.
- Quality is measured mainly with automatic metrics (CLIP, LPIPS, BERTScore) without human evaluation.
Experimental setup
Datasets: COCO SUN
Benchmarks: COCO LSUN
Metrics: CLIP similarity LPIPS similarity LPIPS diversity BERTScore precision BERTScore recall BERTScore F1
Baselines: PEZ LDM+adam LDM+all t
Builds on: Textual Inversion PEZ Latent Diffusion Models Projected gradient descent with delayed projections
Closest papers by meaning
- On Discrete Prompt Optimization for Diffusion Models (ICML 2024 · similarity 0.93)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.92)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.92)
- Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models (ICLR 2025 · similarity 0.91)
- PromptCoT: Align Prompt Distribution via Adapted Chain-of-Thought (CVPR 2024 · similarity 0.91)
- Textualize Visual Prompt for Image Editing via Diffusion Bridge (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.