Research map / Image generation / Text-to-image diffusion
DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes SDXL end-to-end for text-to-design image generation using character-level prompt enhancement, a character localization loss on cross-attention, and self-play DPO on a 1M-image design dataset.
Problem
Generating design images from text prompts requires accurate visual text rendering and style-consistent integration of text and imagery, which prior methods handle with predefined text regions or separate two-stage pipelines.
Approach
An end-to-end one-stage diffusion framework that appends per-character trainable tokens to the CLIP prompt, supervises cross-attention with a character localization loss, and refines text accuracy with self-play DPO using winning ground-truth images and losing model samples selected by OCR accuracy.
Key results
- DesignDiffusion achieves FID 19.87 versus 35.28 for FLUX.1 and 45.10 for SDXL.
- Text accuracy 0.631 and F-score 0.862, above TextDiffuser-2 (0.583 accuracy, 0.827 F-score).
- User study: DesignDiffusion preferred for text quality (0.51) and layout aesthetics (0.54), while SDXL slightly wins image aesthetics (0.43 vs 0.40).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune SDXL on the design image dataset (ablation) | SDXL · DesignDiffusion test set | FID: 45.10 -> 22.73 | Helped |
| Add character-level prompt enhancement (ablation) | Fine-tuned SDXL · DesignDiffusion test set | Text Accuracy: 0.388 -> 0.572 | Helped |
| Add character localization loss (ablation) | Fine-tuned SDXL with prompt enhancement · DesignDiffusion test set | Text Accuracy: 0.572 -> 0.616 | Helped |
| Add self-play DPO fine-tuning (ablation) | Fine-tuned SDXL with prompt enhancement and Lchar · DesignDiffusion test set | Text Accuracy: 0.616 -> 0.631 | Helped |
| Add self-play DPO fine-tuning (ablation) | Fine-tuned SDXL with prompt enhancement and Lchar · DesignDiffusion test set | FID: 19.29 -> 19.87 | Hurt |
Surprising findings
- SP-DPO slightly worsened FID (19.29 to 19.87) while improving text metrics, which the authors attribute to losing-sample selection based only on text accuracy.
Limitations
- Image aesthetics slightly below SDXL in the user study (0.40 vs 0.43).
- SP-DPO slightly increases FID, tied to losing-sample selection criteria based only on text accuracy.
- Relies on a private 1M-image design dataset that is not released.
Experimental setup
Datasets: in-house design image dataset (1M images) character segmentation model outputs
Benchmarks: 5k held-out design image test set
Metrics: FID OCR text precision OCR text recall OCR text F-score OCR text accuracy user preference ratio
Baselines: SDXL SD3 FLUX DeepFloyd-IF GlyphControl TextDiffuser TextDiffuser-2 AnyText
Builds on: Stable Diffusion XL Diffusion-DPO DPOK DDPO
Closest papers by meaning
- Layout-Agnostic Scene Text Image Synthesis with Diffusion Models (CVPR 2024 · similarity 0.89)
- Generating Illustrated Instructions (CVPR 2024 · similarity 0.89)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.88)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.88)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.88)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.