Research map / Image generation / Text-to-image diffusion
Tailored Visions: Enhancing Text-to-Image Generation with Personalized Prompt Rewriting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 300k-prompt personalized text-to-image dataset from 3,115 users and uses retrieved user history plus ChatGPT in-context rewriting to personalize prompts for Stable Diffusion v1-5, evaluated offline and in an online save-rate test.
Problem
Text-to-image users struggle to write prompts that match their personal preferences, and existing prompt rewriting methods ignore user interaction history; no public personalized T2I prompt dataset existed.
Approach
Retrieve the user's most relevant historical prompts (dense CLIP-embedding or BM25) and feed them with the current prompt to an LLM (ChatGPT) that rewrites the prompt to reflect the user's preferences, with one-shot in-context demonstrations; also proposes offline metrics (PMS, Image-Align, ROUGE-L) that recover original prompts from ChatGPT-shortened versions.
Key results
- Personalized PR with EBR retrieval and 1-shot ICL reaches PMS 0.6179, Image-Align 0.6796 and ROUGE-L 0.4686, versus 0.5567/0.6272/0.3268 for shortened prompts.
- Online single-blind test with 247 users and 905 images shows a 17.1% increase in Save actions for personalized prompts over original prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add personalized rewriting using user history | Text-to-image prompt generation · PIP test set (short sentence) | PMS: 0.5567 -> 0.6179 | Helped |
| Add personalized rewriting using user history | Text-to-image prompt generation · PIP test set (short sentence) | Image-Align: 0.6272 -> 0.6796 | Helped |
| Add personalized rewriting using user history | Text-to-image prompt generation · PIP test set (short sentence) | ROUGE-L: 0.3268 -> 0.4686 | Helped |
| Use personalized prompts instead of original prompts | Online text-to-image generation · Online user evaluation | Save actions: 17.1% increase | Helped |
| Use top-1 rather than top-3 retrieved prompts (ablation) | Personalized PR + ICL (EBR) · PIP test set (short sentence) | PMS: 0.6179 -> 0.6057 | Hurt |
| Use top-1 rather than top-3 retrieved prompts (ablation) | Personalized PR + ICL (EBR) · PIP test set (short sentence) | Image-Align: 0.6796 -> 0.6751 | Hurt |
| Use top-1 rather than top-3 retrieved prompts (ablation) | Personalized PR + ICL (EBR) · PIP test set (short sentence) | ROUGE-L: 0.4686 -> 0.4539 | Hurt |
| Use top-5 rather than top-3 retrieved prompts (ablation) | Personalized PR + ICL (EBR) · PIP test set (short sentence) | PMS: 0.6179 -> 0.6204 | Helped |
| Use top-5 rather than top-3 retrieved prompts (ablation) | Personalized PR + ICL (EBR) · PIP test set (short sentence) | ROUGE-L: 0.4686 -> 0.4474 | Hurt |
| Use top-7 rather than top-3 retrieved prompts (ablation) | Personalized PR + ICL (EBR) · PIP test set (short sentence) | PMS: 0.6179 -> 0.6265 | Helped |
| Use top-7 rather than top-3 retrieved prompts (ablation) | Personalized PR + ICL (EBR) · PIP test set (short sentence) | Image-Align: 0.6796 -> 0.6651 | Hurt |
| Use top-7 rather than top-3 retrieved prompts (ablation) | Personalized PR + ICL (EBR) · PIP test set (short sentence) | ROUGE-L: 0.4686 -> 0.4592 | Hurt |
Surprising findings
- General ChatGPT rewriting and Promptist fail to beat shortened prompts on ROUGE-L, with General PR dropping to 0.2082, indicating rewrites drift from user intent without history.
Limitations
- Limited access to personal user information (age, gender) restricts personalization.
- Offline evaluation uses ChatGPT-shortened prompts as proxy inputs, which may not reflect real user queries.
- Online test is small (247 users) and relies on single-blind save actions.
Experimental setup
Datasets: PIP (Personalized Image-Prompt) dataset
Benchmarks: PIP
Metrics: PMS Image-Align ROUGE-L online save rate
Baselines: Promptist General PR (ChatGPT without history) Shortened Prompt
Builds on: Promptist Stable Diffusion CLIP BM25
Closest papers by meaning
- A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis (AAAI 2024 · similarity 0.89)
- AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation (NeurIPS 2024 · similarity 0.88)
- PromptCoT: Align Prompt Distribution via Adapted Chain-of-Thought (CVPR 2024 · similarity 0.88)
- Yo'Chameleon: Personalized Vision and Language Generation (CVPR 2025 · similarity 0.88)
- Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models (ICLR 2025 · similarity 0.88)
- Imagine Before You Draw: Visual Prompt Engineering for Image Generation (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.