Research map / Image generation / Text-to-image diffusion
CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-step
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoT-Diff, which couples an MLLM (Gemini 2.5 Pro) with FLUX.1-schnell so the MLLM refines a 3D scene layout during denoising, injecting semantic and depth conditions via LoRA and condition-aware attention.
Problem
Text-to-image diffusion models struggle to align spatial composition with text in complex scenes, and layout-based methods decouple layout planning from generation so layouts cannot be refined during synthesis.
Approach
Interleaves MLLM-driven 3D layout planning and refinement with the denoising trajectory: at each step the MLLM evaluates the predicted clean image, updates entity attributes, and the updated layout is converted into semantic-layout and depth conditions injected via LoRA branches and a condition-aware attention mask.
Key results
- On 3DSceneBench, CoT-Diff reaches 50.8 on Complex and 48.7 on Multi Relation, versus 40.6 and 34.8 for the best baseline (EliGen).
- Spatial success rate on 50 complex 3DSceneBench prompts is 75.4% for CoT-Diff versus 53.2% for EliGen and 41.6% for FLUX.
- On DVMP, CoT-Diff achieves 80.9 single-object and 78.8 multi-object scores, the highest among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add semantic layout guidance to FLUX (ablation) | FLUX · 3DSceneBench Multi Rela | spatial alignment accuracy: 35.2 -> 40.1 (+4.9) | Helped |
| add semantic layout guidance to FLUX (ablation) | FLUX · 3DSceneBench Complex | spatial alignment accuracy: 40.5 -> 43.7 (+3.2) | Helped |
| add depth map condition to semantic-layout variant (ablation) | FLUX with Semantic Layout · 3DSceneBench Multi Rela | spatial alignment accuracy: 40.1 -> 41.7 (+1.6) | Helped |
| add depth map condition to semantic-layout variant (ablation) | FLUX with Semantic Layout · 3DSceneBench Complex | spatial alignment accuracy: 43.7 -> 48.5 (+4.8) | Helped |
| add optimization to semantic-layout and depth-map variant (ablation) | FLUX with semantic layout and depth map · 3DSceneBench Complex | spatial alignment accuracy: 48.5 -> 50.8 | Helped |
| use Gemini 2.5 Pro instead of GPT-4o for MLLM planning (ablation) | CoT-Diff · 3DSceneBench Front | spatial alignment accuracy: 53.2 -> 54.9 | Helped |
| use Gemini 2.5 Pro instead of GPT-4o for MLLM planning (ablation) | CoT-Diff · 3DSceneBench Behind | spatial alignment accuracy: 53.3 -> 55.2 | Helped |
| use CoT-Diff rather than EliGen | T2I generation · T2I-CompBench Spatial | alignment score: +2.2% | Helped |
| use CoT-Diff rather than EliGen | T2I generation · T2I-CompBench Complex | alignment score: +6.7% | Helped |
| use CoT-Diff rather than the strongest baseline | T2I generation · 3DSceneBench spatial relations | spatial alignment accuracy: +10.2% to +22.4% | Helped |
| use CoT-Diff rather than FLUX | complex scene generation · 50 prompts from 3DSceneBench | spatial success rate: 41.6% -> 75.4% | Helped |
Surprising findings
- Ablation shows adding the semantic layout alone lowers Front relation accuracy relative to FLUX (44.5 vs 48.0).
Limitations
- Reliance on an MLLM at each denoising step introduces non-negligible inference cost.
- Layout planning depends fully on the MLLM; the generator has no internal spatial reasoning.
- 3DSceneBench is built from GPT-4o-generated prompts with limited object set and is evaluated with UniDet detection, which may be a noisy metric.
Experimental setup
Datasets: EliGen LooseControl T2I-CompBench DVMP
Benchmarks: 3DSceneBench DVMP T2I-CompBench
Metrics: UniDet-based spatial relation accuracy GPT-4o alignment score (0-100) T2I-CompBench scores spatial success rate 3D consistency score
Baselines: SD1.5 SDXL PixArt FLUX-schnell RPG EliGen LooseControl
Builds on: FLUX.1-schnell EliGen RPG LooseControl Depth Pro
Closest papers by meaning
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.90)
- Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning (NeurIPS 2025 · similarity 0.89)
- SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation (AAAI 2024 · similarity 0.88)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.88)
- SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation (CVPR 2025 · similarity 0.87)
- Disentangled 3D Scene Generation with Layout Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.