Research map / Image generation / Text-to-image diffusion
Learning to Sample Effective and Diverse Prompts for Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PAG, which reframes text-to-image prompt adaptation as GFlowNet sampling and adds flow reactivation, reward-prioritized replay, and step-wise reward decomposition to avoid mode collapse.
Problem
Prompt adaptation for text-to-image diffusion models with black-box rewards (aesthetics, relevance) tends to collapse to deterministic, similar prompts when optimized with RL.
Approach
Treat prompt adaptation as sampling from an unnormalized reward density via GFlowNets (FL-DB objective), and counter the progressive loss of neural plasticity with periodic last-layer flow reset, reward-prioritized replay, and per-token reward decomposition.
Key results
- On COCO, PAG reaches reward 0.88 and diversity 0.32 in the ablation table, versus Promptist's diversity collapse (0.02 diversity under ImageReward setting).
- Zero-shot transfer from SD v1.4 to SD3 yields reward 0.95 for PAG versus 0.77 for SFT and 0.73 for Promptist.
- Under ImageReward, PAG attains reward 0.83 and diversity 0.29 compared with GFlowNets at 0.81 and 0.20.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PAG instead of SFT | GPT-2 prompt adaptation · COCO, ImageReward | Reward: 0.66 ± 0.01 -> 0.83 ± 0.01 | Helped |
| Use PAG instead of SFT | GPT-2 prompt adaptation · COCO, ImageReward | Diversity: 0.14 ± 0.00 -> 0.29 ± 0.00 | Helped |
| Use PAG instead of SFT | GPT-2 prompt adaptation · COCO, HPSv2 | Diversity: 0.14 ± 0.00 -> 0.37 ± 0.00 | Helped |
| Use PAG rather than Rule-Based prompt adaptation | ChatGPT prompts evaluated with SD v1.5 · ChatGPT prompts, SD v1.5 | Reward: 0.85 ± 0.02 -> 0.87 ± 0.02 | Helped |
| Use PAG rather than Rule-Based prompt adaptation | ChatGPT prompts evaluated with SSD-1B · ChatGPT prompts, SSD-1B | Reward: 0.54 ± 0.05 -> 0.61 ± 0.04 | Helped |
| Add flow reactivation (ablation) | PAG · COCO | Reward: 0.70 ± 0.00 -> 0.88 ± 0.02 | Helped |
| Add flow reactivation (ablation) | PAG · ChatGPT | Reward: 0.81 ± 0.05 -> 0.88 ± 0.04 | Helped |
| Add reward-prioritized sampling (ablation) | PAG · COCO | Reward: 0.30 ± 0.01 -> 0.88 ± 0.02 | Helped |
| Add reward-prioritized sampling (ablation) | PAG · ChatGPT | Diversity: 0.23 ± 0.00 -> 0.32 ± 0.00 | Helped |
| Add reward decomposition (FL) (ablation) | PAG · COCO | Diversity: 0.16 ± 0.00 -> 0.32 ± 0.00 | Helped |
| Add reward decomposition (FL) (ablation) | PAG · ChatGPT | Diversity: 0.18 ± 0.00 -> 0.32 ± 0.00 | Helped |
Surprising findings
- Naive GFlowNet fine-tuning shows a steadily increasing dormant-neuron ratio and decreasing prompt diversity during training.
- Vanilla GFlowNets and Promptist on the ChatGPT brief-prompt dataset underperform simple rule-based postfix heuristics due to mode collapse.
Limitations
- Evaluation is limited to aesthetic and CLIP-based or preference reward proxies without human evaluation.
- DPO-Diff is not directly comparable, so it is discussed only in the appendix.
Experimental setup
Datasets: Lexica DiffusionDB COCO ChatGPT-generated brief prompts
Benchmarks: COCO ChatGPT brief prompts Lexica DiffusionDB
Metrics: reward (aesthetic + CLIP relevance) diversity (average pairwise cosine distance) ImageReward HPSv2
Baselines: Promptist SFT Rule-Based GFlowNets (TB/VarGrad) DPO-Diff DDPO
Builds on: GFlowNets Promptist trajectory balance forward-looking GFlowNets network reset for plasticity
Closest papers by meaning
- Dynamic Prompt Optimizing for Text-to-Image Generation (CVPR 2024 · similarity 0.91)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.91)
- A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis (AAAI 2024 · similarity 0.90)
- Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards (CVPR 2025 · similarity 0.89)
- Subject-driven Text-to-Image Generation via Preference-based Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- PromptCoT: Align Prompt Distribution via Adapted Chain-of-Thought (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.