Research map / Image generation / Text-to-image diffusion
Layout-to-image control
Control spatial arrangement with layouts, instance maps, and spatio-textual representations.
164 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Text-to-image diffusion, keyword cluster “synthesis · layout · object”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 1 of 1 tracked benchmark.
36 papers in 2023–24 and 125 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 9.2% to 14.7%.
Open questions the data points to
- ADE20K (mIoU): best went from 50.7 (up to 2024) to 54.46 in 2025 — Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control (NeurIPS 2025 · 0 citations)
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis (CVPR 2018 · 363 citations)
- 2020 · CookGAN: Causality Based Text-to-Image Synthesis (CVPR 2020 · 76 citations)
- 2021 · Benchmark for Compositional Text-to-Image Synthesis (NeurIPS 2021 · 26 citations)
- 2023 · SpaText: Spatio-Textual Representation for Controllable Image Generation (CVPR 2023 · 156 citations)
- 2024 · MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis (CVPR 2024 · 68 citations)
- 2025 · OmniGen: Unified Image Generation (CVPR 2025 · 56 citations)
- 2026 · Talk2Image: A Multi-Agent System for Multi-Turn Image Generation and Editing (AAAI 2026 · 3 citations)
Earliest papers here
- Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis (CVPR 2018 · 363 citations)
- CookGAN: Causality Based Text-to-Image Synthesis (CVPR 2020 · 76 citations)
- Benchmark for Compositional Text-to-Image Synthesis (NeurIPS 2021 · 26 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
ADE20K · mIoU (higher is better)
4 papers, 2023–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 41.9 | Freestyle Layout-to-Image Synthesis (CVPR 2023 · 68 citations) |
| 2024 | 50.7 | PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis (CVPR 2024 · 18 citations) |
| 2025 | 54.46 | Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control (NeurIPS 2025 · 0 citations) |
What has been tried
78 papers here report 773 measured changes (a component added, removed or swapped, compared with a baseline): helped 80%, no clear effect 5%, hurt 13%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add Shading Aggregation Controller (SAC) | COCO-Position · AP | 52.05 -> 54.69 | Helped | MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis (CVPR 2024 · 68 citations) |
| Use the modified bidirectional attention within each image sequence | Image generation · visual quality | Helped | OmniGen: Unified Image Generation (CVPR 2025 · 56 citations) | |
| Add the visual-language context module | StorySalon test set · FID | 38.81 -> 33.90 | Helped | Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion Models (CVPR 2024 · 42 citations) |
| Add the projection layer | Teddybear and Hydrant, single-image reconstruction · PSNR | 16.55 -> 22.24 | Helped | ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models (CVPR 2024 · 37 citations) |
| add 2D diffusion guidance | Text-to-4D user study · text alignment preference | 11.67% -> 88.33% | Helped | A Unified Approach for Text- and Image-guided 4D Scene Generation (CVPR 2024 · 26 citations) |
| Add adaptive viewpoint sampling | Corgi generation · Geometry consistency | Helped | DreamControl: Control-Based Text-to-3D Generation with 3D Self-Prior (CVPR 2024 · 26 citations) | |
| Add visual-prompted score distillation sampling | Text-to-3D generation · Generation quality | Helped | VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation (CVPR 2024 · 23 citations) | |
| Train with joint volume sampling of albedo, roughness, and metallic strengths rather than axis-only sampling | Joint albedo, roughness, and metallic editing · Qualitative attribute composition | Helped | Alchemist: Parametric Control of Material Properties with Diffusion Models (CVPR 2024 · 19 citations) | |
| Add the layout control map (LCM) | ADE20K · mIoU | 43.5 -> 48.6 | Helped | PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis (CVPR 2024 · 18 citations) |
| Fine-tune SDXL on the design image dataset | DesignDiffusion test set · FID | 45.10 -> 22.73 | Helped | DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Models (CVPR 2025 · 16 citations) |
| add map-guided strategy to image-guided ControlNet | COCO-Thing-Stuff · FID | 28.41 -> 25.12 | Helped | SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation (AAAI 2024 · 15 citations) |
| add global condition | LAVAL indoor HDR · FID | 40.08 -> 35.6 | Helped | Autoregressive Omni-Aware Outpainting for Open-Vocabulary 360-Degree Image Generation (AAAI 2024 · 14 citations) |
What reviewers will expect
Datasets and baselines named in the 58 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- COCO · 5 of 58 papers
- MagicBrush · 4 of 58 papers
- Objaverse · 3 of 58 papers
- T2I-CompBench · 3 of 58 papers
- InstructPix2Pix · 3 of 58 papers
- MSCOCO · 2 of 58 papers
- 3D-FRONT · 2 of 58 papers
- CC12M · 2 of 58 papers
Baselines
- SDXL · 10 of 58 papers
- ControlNet · 6 of 58 papers
- DALL-E 3 · 6 of 58 papers
- GLIGEN · 5 of 58 papers
- GPT-4o · 5 of 58 papers
- InstructPix2Pix · 5 of 58 papers
- AnyText · 4 of 58 papers
- MagicBrush · 4 of 58 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 161 papers; from OpenAlex and the papers' affiliations).
- Zhejiang University · 10 papers
- The Chinese University of Hong Kong · 7 papers
- Tencent · 6 papers
- Shanghai Jiao Tong University · 6 papers
- National University of Singapore · 6 papers
- Adobe · 5 papers
- University of Science and Technology of China · 5 papers
- Alibaba Group · 5 papers
- Huawei · 5 papers
- Peking University · 5 papers
Use it
Download BibTeX of 30 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Compositional text-to-image guidance (194 papers)
- Text-driven synthesis and semantic guidance (137 papers)
- Personalized and region-controlled generation (136 papers)
- Multi-subject personalization (82 papers)
- Text generation and structured prediction (65 papers)
Elsewhere in the topic
- T2I evaluation and controllable adapters (149 papers)
- Inference-time T2I improvement (117 papers)
- Preference optimization for diffusion (82 papers)
- Concept erasure in diffusion models (76 papers)
- Multi-concept composition (56 papers)
- Fairness and bias in generative models (52 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).