Research map / Image generation / Text-to-image diffusion
Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a regional cross-attention module with region reorganization and box-indicator sequenced grounding encoding for rich-context layout-to-image generation, plus two open-vocabulary L2I metrics (CropCLIP and SAMIoU) validated by a user study.
Problem
Open-vocabulary layout-to-image (L2I) generation struggles with lengthy, complex per-object descriptions, and existing evaluations rely on closed-set classifiers/detectors that cannot handle rich-context text.
Approach
Replace self-attention-based object conditioning with regional cross-attention: the image is partitioned into mutually exclusive regions (single object, overlap, background), each region cross-attends to pre-pooled token embeddings of its overlapping object descriptions (with box coordinates as indicators), and background uses a learned null token.
Key results
- On RC CC3M at 512x512 with SDXL, the proposed method reaches CropCLIP 29.42 and SAMIoU 86.56 versus InstDiff's 28.46 and 85.59.
- Filtering objects under 5% or over 50% of image area raised Pearson correlation with user scores from 0.33 to 0.59 for CropCLIP and from 0.15 to 0.52 for SAMIoU.
- Ours (SD1.5) reaches FID 25.81 on RC COCO, the best among compared methods, while matching InstDiff's performance at half the layout-conditioning cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace self-attention conditioning with regional cross-attention | Layout-to-image generation with SD1.5 · RC CC3M evaluation set | CropCLIP: 28.36 -> 28.94 | Helped |
| replace self-attention conditioning with regional cross-attention | Layout-to-image generation with SD1.5 · RC CC3M evaluation set | SAMIoU: 85.58 -> 86.91 | Helped |
| use rich-context descriptions instead of word/phrase descriptions (ablation) | Ours cross-attention with SDXL · RC CC3M evaluation set | CropCLIP: 25.40 -> 29.79 | Helped |
| use rich-context descriptions instead of word/phrase descriptions (ablation) | Ours cross-attention with SDXL · RC CC3M evaluation set | SAMIoU: 86.76 -> 88.10 | Helped |
| add region reorganization (ablation) | Regional cross-attention with SDXL · RC CC3M | CropCLIP: 25.32 -> 28.93 | Helped |
| add region reorganization (ablation) | Regional cross-attention with SDXL · RC CC3M | SAMIoU: 76.92 -> 85.02 | Helped |
| add bounding box indicators (ablation) | Regional cross-attention with SDXL and region reorganization · RC CC3M | SAMIoU: 85.02 -> 86.56 | Helped |
| sample at higher resolution (ablation) | Regional cross-attention with SDXL and box indicators · RC CC3M | CropCLIP: 29.42 -> 29.79 | Helped |
| sample at higher resolution (ablation) | Regional cross-attention with SDXL and box indicators · RC CC3M | SAMIoU: 86.56 -> 88.10 | Helped |
| filter objects smaller than 5% or larger than 50% of the image | Automatic evaluation metrics · RC CC3M user study | CropCLIP Pearson correlation with user scores: 0.33 to 0.59 | Helped |
| filter objects smaller than 5% or larger than 50% of the image | Automatic evaluation metrics · RC CC3M user study | SAMIoU Pearson correlation with user scores: 0.15 to 0.52 | Helped |
Surprising findings
- Automatic metrics were unreliable for very small or very large objects, requiring size-based filtering to correlate with humans.
- Self-attention-based methods did not improve much even when trained on the rich-context dataset, suggesting the attention module, not only data, drives the gains.
Limitations
- Synthetic labels from a vision-language model may be incorrect, which can hurt performance.
- Training images from public datasets are often low quality, leading to artifacts such as watermarks.
- Automatic metrics fail to match human judgment for very small or very large objects, so those were filtered out.
Experimental setup
Datasets: CC3M COCO Stuff RC CC3M RC COCO
Benchmarks: RC COCO RC CC3M
Metrics: CropCLIP SAMIoU FID LPIPS Inception Score Pearson correlation with user scores
Baselines: GLIGEN InstDiff BoxDiff R&B Multi-Instance Sampling
Builds on: GLIGEN InstDiff Stable Diffusion XL Stable Diffusion 1.5 Multi-Instance Sampling
Closest papers by meaning
- InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention (NeurIPS 2025 · similarity 0.91)
- SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation (AAAI 2024 · similarity 0.90)
- OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps (NeurIPS 2025 · similarity 0.89)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.89)
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.89)
- OverLay++: Dense-Overlap Layout-to-Image Generation Dataset (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.