Research map / Image generation / Autoregressive and latent generation
OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OverLayScore, a metric for layout difficulty from bounding-box overlap weighted by semantic similarity, the OverLayBench benchmark balanced across difficulty levels, and CreatiLayout-AM, a CreatiLayout variant fine-tuned with amodal mask attention losses.
Problem
Layout-to-image generation degrades when bounding boxes overlap substantially and the overlapping instances are semantically similar, yet existing benchmarks are skewed toward simple, low-overlap layouts.
Approach
Quantify layout difficulty with OverLayScore (sum of IoU weighted by CLIP caption similarity over overlapping pairs), curate a benchmark balanced across difficulty levels using Qwen2.5-VL grounding with human verification, and fine-tune CreatiLayout with token- and pixel-level attention losses supervised by amodal masks.
Key results
- On OverLayBench-Complex, CreatiLayout-AM-style training improves O-mIoU for EliGen by 1.91% (26.19 to 26.69) and for CreatiLayout it is essentially unchanged (18.05 to 18.07), while on Simple split CreatiLayout-AM raises O-mIoU from 32.52 to 37.69 (+15.90%).
- Existing benchmarks are skewed: most COCO, HiCo and LayoutSAM samples fall in the low OverLayScore regime, whereas OverLayBench is balanced with 2,052 simple, 1,000 regular and 1,000 complex layouts.
- On the Complex split, the best trained method (CreatiLayout-based) reaches mIoU 54.50 and O-mIoU 28.97, and RegionalPrompting leads among training-free methods with mIoU 28.35.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add amodal mask supervision to CreatiLayout (ablation) | CreatiLayout · OverLayBench-Simple | O-mIoU (%): 32.52 -> 37.69; +15.90% | Helped |
| Add amodal mask supervision to CreatiLayout (ablation) | CreatiLayout · OverLayBench-Regular | O-mIoU (%): 20.67 -> 21.79; +5.42% | Helped |
| Add amodal mask supervision to CreatiLayout (ablation) | CreatiLayout · OverLayBench-Complex | O-mIoU (%): 18.05 -> 18.07; +0.11% | Helped |
| Add amodal mask supervision to EliGen (ablation) | EliGen · OverLayBench-Simple | O-mIoU (%): 43.72 -> 46.43; +6.20% | Helped |
| Add amodal mask supervision to EliGen (ablation) | EliGen · OverLayBench-Regular | O-mIoU (%): 32.62 -> 33.85; +3.74% | Helped |
| Add amodal mask supervision to EliGen (ablation) | EliGen · OverLayBench-Complex | O-mIoU (%): 26.19 -> 26.69; +1.91% | Helped |
Surprising findings
- CreatiLayout-AM gives no O-mIoU gain on the Complex split (18.05 to 18.07) and slightly lower mIoU, suggesting amodal supervision does not transfer under distribution shift.
- Training-free RegionalPrompting outperforms several training-based methods on the Simple split in mIoU and O-mIoU.
Limitations
- CreatiLayout-AM gains are largely confined to Simple and Regular splits; Complex split performance is flat or slightly lower.
- Evaluation relies on Qwen2.5-VL-32B as both detector and judge for SR_E and SR_R, which may introduce bias.
- OverLayScore depends on CLIP caption similarity as a proxy for semantic overlap.
Experimental setup
Datasets: COCO LayoutSAM HiCo-7k
Benchmarks: OverLayBench
Metrics: mIoU O-mIoU SR_E SR_R CLIP score FID
Baselines: GLIGEN InstanceDiffusion MIGC HiCo 3DIS EliGen DreamRender CreatiLayout BoxDiff RegionalPrompting
Builds on: CreatiLayout TokenCompose EliGen Flux
Closest papers by meaning
- OverLay++: Dense-Overlap Layout-to-Image Generation Dataset (NeurIPS 2026 · similarity 0.93)
- Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation (NeurIPS 2024 · similarity 0.89)
- InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention (NeurIPS 2025 · similarity 0.89)
- Learning to Generate Multiple Objects from Dense and Occluded Layouts (NeurIPS 2026 · similarity 0.87)
- HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation (NeurIPS 2024 · similarity 0.87)
- Check Locate Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.