Research map / Image generation / Text-to-image diffusion
PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PLACE, a fine-tuning module for Stable Diffusion that fuses a downsampled layout control map with cross-attention via a timestep-adaptive weight, plus Semantic Alignment and Layout-Free Prior Preservation losses, for semantic image synthesis.
Problem
Semantic image synthesis with pre-trained text-to-image models struggles to keep layouts faithful to semantic maps and to preserve visual quality and pretrained priors, since naive resizing of semantic maps loses layout detail and prior cross-attention modifications disrupt global text-image interac
Approach
A layout control map encodes per-token class proportions in each image token's receptive field instead of naive resizing; a learned timestep-dependent scalar blends this layout map with the original cross-attention maps, and auxiliary losses (SA aligns self-attention aggregation with fusion maps; LFP uses layout-free text-image pairs with alpha=0) preserve priors.
Key results
- On ADE20K, PLACE reaches FID 22.3 and mIoU 50.7, outperforming the best prior FID (FreestyleNet 25.0 among diffusion methods; ECGAN 25.8 overall).
- On COCO-Stuff, PLACE achieves FID 14.0 vs. 14.4 for FreestyleNet, the second-best score.
- In out-of-distribution new-object synthesis, PLACE gets mIoU 33.0 vs. 24.6 for FreestyleNet and FID 18.1 vs. 20.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the layout control map (LCM) (ablation) | PLACE baseline · ADE20K | mIoU: 43.5 -> 48.6 | Helped |
| Add the layout control map (LCM) (ablation) | PLACE baseline · New object classes (COCO-Stuff) | mIoU: 25.3 -> 28.5 | Helped |
| Use timestep-adaptive alpha for fusion (ablation) | PLACE baseline · ADE20K | FID: 24.2 -> 23.0 | Helped |
| Use timestep-adaptive alpha for fusion (ablation) | PLACE with LCM · ADE20K | FID: 23.4 -> 22.7 | Helped |
| Add Semantic Alignment (SA) loss (ablation) | PLACE baseline · ADE20K | mIoU: 43.5 -> 46.7 | Helped |
| Add Semantic Alignment (SA) loss (ablation) | PLACE with LCM and adaptive alpha · ADE20K | mIoU: 50.1 -> 50.9 | Helped |
| Add Layout-Free Prior Preservation (LFP) loss (ablation) | PLACE with LCM and adaptive alpha · New object classes (COCO-Stuff) | mIoU: 29.4 -> 32.8 | Helped |
| Add Layout-Free Prior Preservation (LFP) loss (ablation) | PLACE with LCM, adaptive alpha, and SA loss · New object classes (COCO-Stuff) | mIoU: 29.9 -> 33.0 | Helped |
Limitations
- Layout consistency mIoU on ADE20K is only comparable to GAN-based state of the art, not clearly superior (e.g., ECGAN 50.6 vs. 50.7).
- Relies on fine-tuning on limited-scale paired semantic-mask datasets, with the paper noting prior loss risks without LFP.
- Evaluation of out-of-distribution new style/attribute uses only 260 synthesized images and CLIP similarity, a limited protocol.
Experimental setup
Datasets: ADE20K COCO-Stuff OpenImages LAION-5B
Benchmarks: ADE20K COCO-Stuff
Metrics: FID mIoU CLIP text-image similarity
Baselines: pix2pixHD SPADE CC-FPSE OASIS SC-GAN SAFM ECGAN ControlNet T2I-Adapter FreestyleNet PITI SDM
Builds on: Stable Diffusion FreestyleNet ControlNet DreamBooth-style prior preservation
Closest papers by meaning
- SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation (AAAI 2024 · similarity 0.87)
- InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention (NeurIPS 2025 · similarity 0.87)
- Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.87)
- Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation (NeurIPS 2024 · similarity 0.87)
- Control and Realism: Best of Both Worlds in Layout-to-Image without Training (ICML 2025 · similarity 0.87)
- Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.