Research map / Image generation / Text-to-image diffusion
Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RPG, a training-free framework where GPT-4 recaptions prompts, plans regional layouts via chain-of-thought, and SDXL generates each region with complementary regional diffusion, extended to editing in a closed loop.
Problem
Text-to-image diffusion models struggle with complex prompts involving multiple objects, attributes, and relationships, and existing layout/attention methods give rough spatial control while feedback-based methods need costly training.
Approach
Use an MLLM (GPT-4) as a global planner that recaptions subprompts and chooses non-overlapping rectangular regions via chain-of-thought; then generate each region with a separate prompt in parallel within the diffusion denoising loop and merge them by resize-and-concatenate blended with a base-prompt latent, plus contour-based inpainting for closed-loop editing.
Key results
- On T2I-CompBench, RPG reaches 0.8335 color, 0.6801 shape, 0.8129 texture, 0.4547 spatial, and 0.5408 complex, the best among compared methods (SDXL complex 0.4091, ConPreDiff 0.4184).
- The paper reports RPG outperforms SDXL and DALL-E 3 particularly in multi-category object composition and text-image alignment, with qualitative comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RPG instead of ConPreDiff | Text-to-image generation · T2I-CompBench Color | score: 0.7019 -> 0.8335 | Helped |
| Use RPG instead of ConPreDiff | Text-to-image generation · T2I-CompBench Shape | score: 0.5637 -> 0.6801 | Helped |
| Use RPG instead of PixArt-α | Text-to-image generation · T2I-CompBench Texture | score: 0.7044 -> 0.8129 | Helped |
| Use RPG instead of ConPreDiff | Text-to-image generation · T2I-CompBench Spatial Relationship | score: 0.2362 -> 0.4547 | Helped |
| Use RPG instead of ConPreDiff | Text-to-image generation · T2I-CompBench Non-Spatial Relationship | score: 0.3195 -> 0.3462 | Helped |
| Use RPG instead of ConPreDiff | Text-to-image generation · T2I-CompBench Complex | score: 0.4184 -> 0.5408 | Helped |
| Add prompt recaptioning (ablation) | RPG · Text-to-image generation | key-word inclusion and image precision | Helped |
| Add CoT planning (ablation) | RPG · Complex text-to-image generation | relationship and attribute expression | Helped |
| Use a proper base-prompt ratio (ablation) | RPG complementary regional diffusion · Text-to-image generation | composition coherence | Helped |
| Increase the base-prompt ratio excessively (ablation) | RPG complementary regional diffusion · Text-to-image generation | composition quality | Hurt |
| Increase the number of regional-diffusion hierarchies (ablation) | RPG · Text-to-image generation | generation quality | Helped |
| Run closed-loop self-refinement over multiple editing rounds (ablation) | RPG image editing · Text-guided image editing | editing precision | Helped |
Surprising findings
- SDXL and DALL-E 3 perform poorly on numeric accuracy and complex relationships in the authors' qualitative tests.
Limitations
- Relies on closed-source GPT-4 for planning, making results dependent on an API.
- Excessive base-prompt ratio causes confusion between base and regional prompts; base ratio must be tuned per prompt type.
- Layout-based planning can still fail on numeric or relational cases, and closed-loop refinement is capped for time efficiency.
Experimental setup
Datasets: T2I-CompBench
Benchmarks: T2I-CompBench
Metrics: Color Shape Texture Spatial Non-Spatial Complex
Baselines: SDXL DALL-E 3 LMD+ ConPreDiff PixArt-alpha GORS Structured Diffusion Attn-Exct v2 InstructPix2Pix MasaCtrl Prompt2Prompt
Builds on: Stable Diffusion XL Composable Diffusion LMD ControlNet Prompt-to-Prompt
Closest papers by meaning
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.92)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.91)
- LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation (AAAI 2025 · similarity 0.90)
- DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation (ICLR 2025 · similarity 0.90)
- Progressive Text-to-Image Diffusion with Soft Latent Direction (AAAI 2024 · similarity 0.90)
- Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.