Research map / Image generation / Text-to-image diffusion
Dynamic Prompt Optimizing for Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prompt Auto-Editing (PAE), which fine-tunes GPT-2 on filtered prompt logs and then uses PPO to learn dynamic fine-control prompts (tokens with effect time ranges and weights) for Stable Diffusion v1.4.
Problem
Text-to-image diffusion models are highly sensitive to prompt wording, and manually tuning prompt modifiers, weights, and injection time steps is costly and labor-intensive.
Approach
Formulates prompt editing as an RL problem where a policy appends modifier tokens, each paired with a denoising-step effect range and a weight (DF-Prompt), trained with PPO against a multi-part reward (CLIP score, PickScore, aesthetic score), after an SFT stage on automatically filtered prompt-image data.
Key results
- On Lexica.art, PAE reaches 73.9% PickScore win rate and 6.12 aesthetic score, versus 68.4% and 6.11 for Promptist.
- On COCO (out-of-domain), PAE achieves 53.8% PickScore vs 53.4% for Promptist, with aesthetic score 6.09 vs 6.15.
- CMMD on COCO improves from 1.147 (Promptist) to 1.125 (PAE).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply PAE prompt editing | Stable Diffusion v1.4 · Lexica.art | PickScore: 72.5% -> 73.9% | Helped |
| apply PAE prompt editing | Stable Diffusion v1.4 · DiffusionDB | Aesthetic Score: 6.06 -> 6.07 | Helped |
| apply PAE prompt editing | Stable Diffusion v1.4 · COCO | Aesthetic Score: 6.15 -> 6.09 | Hurt |
| use σ = 0.5 instead of σ = 1.0 for EReP data settings (ablation) | EReP · DiffusionDB validation | Aesthetic Score: 5.95 -> 6.01; 5.94 -> 6.03 | Helped |
| set γ = 0.01 instead of γ = 0.00 with σ = 0.5 (ablation) | EReP · DiffusionDB validation | Aesthetic Score: 6.01 -> 6.03 | Helped |
| set γ = 0.01 instead of γ = 0.00 with σ = 1.0 (ablation) | EReP · DiffusionDB validation | Aesthetic Score: 5.95 -> 5.94 | Hurt |
| set β = 1 instead of β = 0 (ablation) | EDFP · DiffusionDB validation | PickScore: 58.0% -> 9.6% | Hurt |
| set β = 1 instead of β = 0 (ablation) | EDFP · DiffusionDB validation | CLIP Score: 0.26 -> 0.28 | Helped |
| increase α from 1 to 5 with β = 1, κ = 18 (ablation) | EDFP · DiffusionDB validation | Aesthetic Score: 5.54 -> 5.97 | Helped |
| increase α from 1 to 5 with β = 1, κ = 18 (ablation) | EDFP · DiffusionDB validation | CLIP Score: 0.28 -> 0.26 | Hurt |
| increase κ from 16 to 18 with α = 1, β = 0 (ablation) | EDFP · DiffusionDB validation | PickScore: 53.8% -> 58.0% | Helped |
| add second-stage online reinforcement learning (ablation) | EReP · DiffusionDB validation | PickScore: 53.8% -> 57.8% | Helped |
Surprising findings
- Reward-driven policy prefers lower weight (0.75) over higher weight (1.25) for complex modifier terms, and injecting texture/style terms in the latter half of denoising works better than the first half.
Limitations
- Gains over Promptist are small on some metrics (e.g., COCO aesthetic score lower than Promptist's)
- Evaluated only with Stable Diffusion v1.4 and a single reward configuration
- Relies on a fixed set of 15 possible modifier time ranges and 5 weight values
Experimental setup
Datasets: Lexica.art DiffusionDB COCO
Benchmarks: Lexica.art DiffusionDB COCO
Metrics: aesthetic score CLIP score PickScore CMMD
Baselines: Promptist GPT-2 human-written prompts short prompt
Builds on: Promptist PPO GPT-2 Stable Diffusion CLIP
Closest papers by meaning
- Learning to Sample Effective and Diverse Prompts for Text-to-Image Generation (CVPR 2025 · similarity 0.91)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.89)
- A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis (AAAI 2024 · similarity 0.89)
- Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation (ICLR 2024 · similarity 0.89)
- Textualize Visual Prompt for Image Editing via Diffusion Bridge (AAAI 2025 · similarity 0.88)
- Rich Human Feedback for Text-to-Image Generation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.