Research map / Image generation / Text-to-image diffusion
A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a coarse-fine prompt triplet dataset (CFP) and trains a CLIP-to-T5 prompt refiner (UF-FGTG) supervised by Stable Diffusion loss, CLIP-style alignment and an adaptive image-feature module to rewrite novice prompts into model-preferred prompts.
Problem
Novice users enter short, coarse prompts that differ in length and style from the long, fine-grained prompts text-to-image models were trained on, so they struggle to get desired images; existing prompt rewriting is unimodal and not tied to the image model.
Approach
Train a prompt refiner (CLIP text encoder plus T5 decoder) to map coarse prompts into the fine-grained prompt feature space, using Stable Diffusion's denoising loss as image-related supervision and an adaptive image-feature extraction module with a CLIP-based contrastive loss to preserve diversity.
Key results
- UF-FGTG achieves NIMA-TID 5.73, MUSIQ-KonIQ 69.74, DB-CNN 65.21, TReS 83.34, NIMA-AVA 5.48, MUSIQ-AVA 5.97, beating GPT-4 and GPT-2* baselines across all six metrics.
- The paper reports an average improvement of about 5% across six quality and aesthetic metrics.
- Removing the adaptive feature extraction module (Lclip) lowers scores, e.g. NIMA-AVA from 5.48 to 5.41 and DB-CNN from 65.21 to 64.74.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the prompt refiner with UF-FGTG, including image-related loss and adaptive feature extraction | Text prompt generation for Stable Diffusion-v2.1 · CFP test prompts | NIMA-TID: 5.59 -> 5.73 | Helped |
| Train the prompt refiner with UF-FGTG, including image-related loss and adaptive feature extraction | Text prompt generation for Stable Diffusion-v2.1 · CFP test prompts | TReS: 80.90 -> 83.34 | Helped |
| Add Stable Diffusion image-related supervision (Lmse) (ablation) | UF-FGTG · CFP prompts | NIMA-TID: 5.53 -> 5.73 | Helped |
| Add adaptive feature extraction module (Lclip) (ablation) | UF-FGTG · CFP prompts | NIMA-TID: 5.68 -> 5.73 | Helped |
| Add both image-related losses Lmse and Lclip (ablation) | UF-FGTG · CFP prompts | NIMA-TID: 5.48 -> 5.73 | Helped |
| Increase generated prompt length beyond 6 tokens (ablation) | UF-FGTG · CFP prompts, Stable Diffusion-v2.1 images | NIMA-TID | Mixed |
Surprising findings
- Longer prompts do not always improve image quality; the paper finds about 6 added tokens generalize best, attributing decline at longer lengths to overfitting.
Limitations
- Evaluation relies only on non-reference image quality and aesthetic predictors, with no human study.
- Baselines are general LLMs used zero-shot or fine-tuned on CFP; comparisons are limited to a single image generator (Stable Diffusion-v2.1).
Experimental setup
Datasets: CFP (Lexica.art prompts with Stable Diffusion-v2.1 images) DiffusionDB AVA KonIQ-10K TID2013
Benchmarks: CFP test split
Metrics: NIMA-TID MUSIQ-KonIQ DB-CNN TReS NIMA-AVA MUSIQ-AVA
Baselines: GPT-2 FLAN-T5 GPT-3.5 GPT-4
Builds on: Stable Diffusion CLIP FLAN-T5 OpenCLIP DiffusionDB
Closest papers by meaning
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Learning to Sample Effective and Diverse Prompts for Text-to-Image Generation (CVPR 2025 · similarity 0.90)
- PromptCoT: Align Prompt Distribution via Adapted Chain-of-Thought (CVPR 2024 · similarity 0.90)
- Tailored Visions: Enhancing Text-to-Image Generation with Personalized Prompt Rewriting (CVPR 2024 · similarity 0.89)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.89)
- Dynamic Prompt Optimizing for Text-to-Image Generation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.