Research map / Image generation / Text-to-image diffusion
Minority-Focused Text-to-Image Generation via Prompt Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MinorityPrompt, an online token-based prompt optimization that updates a learnable placeholder embedding during diffusion sampling to generate low-likelihood (minority) text-to-image samples while preserving prompt semantics.
Problem
Pretrained text-to-image latent diffusion models, guided by CFG, concentrate on high-density regions, making it hard to generate minority (low-likelihood) samples for a given prompt.
Approach
Instead of optimizing the entire text embedding during inference, append a placeholder token to the prompt and optimize only its embedding at each sampling step with a likelihood-inspired reconstruction objective (non-CFG, with a stop-gradient-free two-term loss), which preserves prompt semantics while pushing samples to low-density regions.
Key results
- On SDv1.5, MinorityPrompt reaches likelihood 0.8971 bpd versus 1.0367 for DDIM and 0.9540 for SGMS, while CLIPScore rises slightly to 31.54 from 31.48.
- On SDXL-Lightning, MinorityPrompt lowers likelihood to 0.5457 bpd (DDIM 0.6082) with CLIPScore 31.34 and ImageReward 0.7098 close to DDIM's 0.7331.
- Ablation on SDv1.5 shows the naive objective (Eq. 6) degrades CLIPScore to 30.30 and ImageReward to -0.19, versus 31.74 and 0.28 for the proposed objective.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MinorityPrompt instead of SGMS | SDv1.5 · MS-COCO validation | Likelihood (bpd): 0.9540 -> 0.8971 | Helped |
| Use MinorityPrompt instead of SGMS | SDv2.0 · MS-COCO validation | Likelihood (bpd): 0.9898 -> 0.9143 | Helped |
| Use MinorityPrompt instead of SGMS | SDXL-Lightning · MS-COCO validation | Likelihood (bpd): 0.5462 -> 0.5457 | Helped |
| Add CFG clean predictions to Eq. (7) (ablation) | MinorityPrompt on SDv1.5 · SDv1.5 evaluation | Likelihood (bpd): 0.9230 -> 0.9399 | Hurt |
| Add stop-gradient to the second clean prediction (ablation) | MinorityPrompt on SDv1.5 · SDv1.5 evaluation | Likelihood (bpd): 0.9230 -> 0.9290 | Hurt |
| Use Cv instead of C in the second clean prediction (ablation) | MinorityPrompt on SDv1.5 · SDv1.5 evaluation | Likelihood (bpd): 0.9230 -> 0.9281 | Hurt |
| Add CFG clean predictions, stop-gradient, and Cv in the second prediction (ablation) | MinorityPrompt on SDv1.5 · SDv1.5 evaluation | Likelihood (bpd): 0.9230 -> 0.9245 | Hurt |
Surprising findings
- The minority sampler often counters demographic biases in T2I models, e.g., changing apparent age or skin color of subjects.
- Optimizing the entire text embedding during inference degrades text-image alignment, motivating optimizing only an appended token.
Limitations
- Minority sampling comes with some loss in image quality, shown by lower PickScore and Density than DDIM.
- Requires per-step optimization with extra network evaluations, adding inference cost.
Experimental setup
Datasets: MS-COCO validation set (10K captions) LAION (pretraining of backbones)
Benchmarks: MS-COCO validation captions
Metrics: CLIPScore PickScore ImageReward Precision Recall Density Coverage log-likelihood (bpd) In-Batch Similarity
Baselines: DDIM DDIM + null prompt CADS SGMS
Builds on: DDIM classifier-free guidance SGMS CADS
Closest papers by meaning
- DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation (ICLR 2025 · similarity 0.92)
- On Discrete Prompt Optimization for Diffusion Models (ICML 2024 · similarity 0.91)
- Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusion (CVPR 2025 · similarity 0.90)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
- Boost-and-Skip: A Simple Guidance-Free Diffusion for Minority Generation (ICML 2025 · similarity 0.90)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.