Research map / Image generation / Text-to-image diffusion
Progressive Text-to-Image Diffusion with Soft Latent Direction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SRF (Stimulus, Response, Fusion), a training-free progressive text-to-image framework where a fine-tuned GPT decomposes long prompts into synthesis/editing/erasing steps executed on Stable Diffusion v1.4 with soft latent guidance and latent fusion.
Problem
Text-to-image diffusion models handle one or two entities well but fail on prompts with many entities and spatial or relational constraints, and existing methods do not unify synthesis and editing in a progressive procedure.
Approach
Decompose a long prompt into atomic synthesis/editing/erasing prompts with an LLM and apply them sequentially; a soft stimulus loss on cross-attention updates the latent (instead of hard attention replacement), and a latent fusion step with a layout mask that shifts to an attention mask blends each new entity into the previous image.
Key results
- Object Recall 64.4 and Relation Accuracy 50.8 versus 40.7/19.8 for Stable Diffusion and 50.3/23.4 for Attend-and-excite.
- In progressive comparison, synthesis object score 74 and relation 60 versus 67/52 for Blended Latent; editing 72 and erasing 50 are best among baselines.
- Ablation: removing latent fusion drops Object Recall to 38.8 and Relation Accuracy to 21.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use progressive synthesis with the proposed framework instead of Attend-and-Excite | Text-to-image generation · 500 GPT prompts | Object Recall: 50.3 -> 64.4 | Helped |
| Use progressive synthesis with the proposed framework instead of Attend-and-Excite | Text-to-image generation · 500 GPT prompts | Relation Accuracy: 23.4 -> 50.8 | Helped |
| Use the proposed progressive framework instead of Blended Latent | Progressive text-to-image generation · SUN (25 images), synthesis | Object rating score: 67 -> 74 | Helped |
| Use the proposed progressive framework instead of Blended Latent | Progressive text-to-image generation · SUN (25 images), synthesis | Relation rating score: 52 -> 60 | Helped |
| Use the proposed progressive framework instead of Blended Latent | Progressive text-to-image generation · SUN (25 images), editing | Editing rating score: 67 -> 72 | Helped |
| Use the proposed progressive framework instead of Blended Latent | Progressive text-to-image generation · SUN (25 images), erasing | Erasing rating score: 46 -> 50 | Helped |
| Add Latent Fusion (ablation) | Progressive text-to-image generation · 500 GPT prompts | Object Recall: 38.8 -> 64.4 | Helped |
| Add Latent Fusion (ablation) | Progressive text-to-image generation · 500 GPT prompts | Relation Accuracy: 21.8 -> 50.8 | Helped |
| Add Stimulus & Response (ablation) | Progressive text-to-image generation · 500 GPT prompts | Object Recall: 58.3 -> 64.4 | Helped |
| Add Stimulus & Response (ablation) | Progressive text-to-image generation · 500 GPT prompts | Relation Accuracy: 45.2 -> 50.8 | Helped |
Surprising findings
- Removing Stimulus & Response only slightly lowers Object Recall (58.3 vs 64.4) but drops Relation Accuracy more sharply (45.2 vs 50.8).
Limitations
- Not all text can be decomposed into a sequence of short prompts (e.g., 'a horse under a car and between a cat and a dog').
- Evaluation is small-scale: 500 prompts for single-stage and 25 images for progressive, with human raters from AMT and 8 human evaluations for relation accuracy.
Experimental setup
Datasets: SUN GPT-constructed text prompts
Benchmarks: 500 GPT-constructed multi-object prompts (single-stage) 25 SUN-based progressive scenarios
Metrics: Object Recall Relation Accuracy rating score (0-2 fidelity, max 100)
Baselines: Stable Diffusion Attend-and-excite Structured Diffusion InstructPix2Pix Stable-inpainting Blended Latent
Builds on: Stable Diffusion Attend-and-Excite Structured Diffusion Prompt-to-Prompt DDIM inversion
Closest papers by meaning
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.91)
- Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.90)
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.90)
- Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.90)
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.90)
- LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.