Research map / Image generation / Text-to-image diffusion
LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses an LLM (ChatGPT) to extract a Scene Blueprint (layouts, per-object descriptions, background prompt) from long prompts, then generates the image in two phases (layout-to-image global scene, then CLIP-guided box-level refinement with paint-by-example composition) to improve faithful object inclusion.
Problem
Text-to-image diffusion models omit or misrepresent objects when given long, detailed prompts describing complex scenes with many objects, partly because the CLIP text encoder truncates inputs at 77 tokens.
Approach
Decompose a long prompt via an LLM into a Scene Blueprint (boxes, object descriptions, background), interpolate multiple LLM-generated layouts for robustness, generate a global layout-conditioned scene, then iteratively refine each box using a reference image from Stable Diffusion, mask-based composition, and a CLIP text+image guidance loss.
Key results
- PAR score of 85% versus 49% for Stable Diffusion, 57% for GLIGEN, and 69% for LayoutGPT on challenging prompts with 3 to 10 objects.
- Compared with DeepFloyd (60% PAR, about 8.33 min), the method reports 85% PAR at about 3.16 min average inference time.
- A 2-AFC user study found a majority of users preferred the method's images for prompt adherence and fidelity over the baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLM Blueprint with Scene Blueprints and iterative refinement instead of Stable Diffusion | Text-to-image generation from complex prompts · Complex prompts | PAR score (%): 49 -> 85 | Helped |
| Use LLM Blueprint with Scene Blueprints and iterative refinement instead of GLIGEN | Text-to-image generation from complex prompts · Complex prompts | PAR score (%): 57 -> 85 | Helped |
| Use LLM Blueprint with Scene Blueprints and iterative refinement instead of LayoutGPT | Text-to-image generation from complex prompts · Complex prompts | PAR score (%): 69 -> 85 | Helped |
| Use LLM Blueprint with Scene Blueprints and iterative refinement instead of DenseDiffusion | Text-to-image generation from complex prompts · Complex prompts | PAR score (%): 52 -> 85 | Helped |
| Use LLM Blueprint with Scene Blueprints and iterative refinement instead of DeepFloyd | Text-to-image generation from complex prompts · Complex prompts | PAR score (%): 60 -> 85 | Helped |
| Add layout interpolation across multiple LLM-generated layouts (ablation) | LLM Blueprint · Complex prompts with spatial object relationships | spatial positioning | Helped |
| Add CLIP multimodal guidance during box-level refinement (ablation) | LLM Blueprint · Box-level object refinement | prompt and prototype adherence | Helped |
| Generate multiple layouts and interpolate them instead of using only one layout (ablation) | LLM Blueprint · Complex prompts with spatial relationships | spatial coherence | Helped |
| Use an unambiguous prompt that explicitly specifies object positions | ChatGPT bounding-box generation · Cat-and-dog spatial prompts | correct box-position frequency | Helped |
Surprising findings
- ChatGPT shows an inherent left/right bias when the spatial relation is ambiguous (e.g., cat and dog sitting one on each side), placing the cat on the right about 70% of the time.
Limitations
- Fixed box layouts during the refinement phase; boxes are not adjusted dynamically.
- Overlapping boxes are handled by sorting by size, and each object is refined in isolation without modeling inter-object relationships.
- LLM layout errors and spatial bias (e.g., ambiguous left/right prompts) require manual interpolation control via eta.
Experimental setup
Datasets: COCO PASCAL
Benchmarks: Prompt Adherence Recall (PAR) on ChatGPT-generated complex prompts and COCO/PASCAL-derived prompts
Metrics: PAR (object recall via off-the-shelf object detector) inference time 2-AFC user preference
Baselines: Stable Diffusion GLIGEN LayoutGPT LLM-Grounded Diffusion DenseDiffusion DeepFloyd
Builds on: LayoutGPT paint-by-example classifier guidance DDIM
Closest papers by meaning
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.92)
- LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation (AAAI 2025 · similarity 0.91)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.91)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.90)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.