Research map / Image generation / Text-to-image diffusion
SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSMG, a map-guided diffusion model for layout-to-image generation that builds a spatial-semantic feature map from instance text embeddings and uses relation-sensitive and location-sensitive attention to control both spatial placement and semantics.
Problem
Layout-to-image generation with diffusion models suffers from weak pixel-level spatial control and limited per-instance semantic control, since token-guided (e.g., GLIGEN, ReCo) and image-guided (ControlNet) methods lose spatial structure or only use global captions.
Approach
Replace token or RGB-image layout guidance with a 2D spatial-semantic feature map populated with per-instance text-encoder embeddings, refine it with relation-sensitive attention among instances and the scene token, and inject it into the U-Net via a gated location-sensitive attention where the map serves as queries over noise features.
Key results
- Our-v21 achieves FID 20.82 vs 21.04 for GLIGEN and 27.47 for ReCo on COCO, the best FID in the table.
- YOLO score mAP improves from 30.5 (ReCo) to 37.6 for Our-v21, and mAP50 reaches 59.0.
- Ablation: map-guided strategy alone lowers FID from 28.41 (ControlNet baseline) to 25.12; adding RSA and LSA gives 20.82.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SSMG map-guided generation instead of token-guided or image-guided baselines | Layout-to-image generation on COCO-Thing-Stuff · COCO-Thing-Stuff | YOLO score (mAP): 30.5 -> 37.6 | Helped |
| add map-guided strategy to image-guided ControlNet (ablation) | SSMG ablation model · COCO-Thing-Stuff | FID: 28.41 -> 25.12 | Helped |
| add map-guided strategy to image-guided ControlNet (ablation) | SSMG ablation model · COCO-Thing-Stuff | YOLO score (mAP): 25.2 -> 30.5 | Helped |
| add RSA to MG (ablation) | SSMG ablation model · COCO-Thing-Stuff | FID: 25.12 -> 21.41 | Helped |
| add RSA to MG (ablation) | SSMG ablation model · COCO-Thing-Stuff | YOLO score (mAP): 30.5 -> 32.7 | Helped |
| add LSA to MG (ablation) | SSMG ablation model · COCO-Thing-Stuff | FID: 25.12 -> 23.49 | Helped |
| add LSA to MG (ablation) | SSMG ablation model · COCO-Thing-Stuff | YOLO score (mAP): 30.5 -> 35.1 | Helped |
| add RSA and LSA to MG (ablation) | SSMG ablation model · COCO-Thing-Stuff | FID: 25.12 -> 20.82 | Helped |
| add RSA and LSA to MG (ablation) | SSMG ablation model · COCO-Thing-Stuff | YOLO score (mAP): 30.5 -> 37.6 | Helped |
Surprising findings
- Our-v15 has the highest IS (33.99) but FID is worse (22.31) than Our-v21, and IS is lower than the Control* baseline's IS would suggest only in some cases; the IS advantage is not uniform across versions.
Limitations
- Relies on a pre-trained caption model to generate per-instance descriptions for COCO crops, so semantic quality depends on that captioner.
- Gains over GLIGEN and ReCo in FID and IS are small, as the paper acknowledges, since generation quality is largely inherited from the T2I backbone.
Experimental setup
Datasets: COCO-Thing-Stuff
Benchmarks: COCO-Thing-Stuff
Metrics: FID IS Diversity Score (LPIPS) YOLO score mAP/mAP50/mAP75
Baselines: LostGAN Context L2I LAMA TwFA ReCo GLIGEN LayoutDiffusion ControlNet
Builds on: Stable Diffusion ControlNet LDM GLIGEN
Closest papers by meaning
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.90)
- Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation (NeurIPS 2024 · similarity 0.90)
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.90)
- Spatial Transport Optimization by Repositioning Attention Map for Training-Free Text-to-Image Synthesis (CVPR 2025 · similarity 0.89)
- SpotActor: Training-Free Layout-Controlled Consistent Image Generation (AAAI 2025 · similarity 0.88)
- CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-step (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.