Research map / Image generation / Text-to-image diffusion
Control and Realism: Best of Both Worlds in Layout-to-Image without Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WinWinLay, a training-free layout-to-image method that adds a non-local attention prior to the attention energy function and a Langevin-dynamics adaptive update on Stable Diffusion 1.5 to improve layout control and realism.
Problem
Training-free layout-to-image generation with pre-trained text-to-image diffusion models suffers from imprecise object localization (attention collapsing to local regions) and out-of-distribution artifacts caused by backpropagation-based latent updates.
Approach
Augments the attention energy function with a non-local attention prior (KL to a center-biased distribution, decayed over denoising steps) to spread attention across the box, and replaces vanilla gradient updates with Langevin dynamics that combine the pre-trained score with the layout energy gradient, using an adaptive weight derived from a two-task Nash bargaining analysis.
Key results
- On COCO2014, WinWinLay reaches AP 19.74 and CLIP-s 0.327 versus 17.63 AP for the best baseline R&B, with FID 26.85 vs 27.51 for AttRe.
- In a user study with 150 participants, 27.7% of its controllability votes and 29.8% of quality votes were favored over baselines (CSG 20.9 and 21.0).
- Ablation on COCO2014 shows non-local attention energy plus adaptive update reaches AP 19.7 and AP50 49.2, above the baseline's AP 8.4 and AP50 25.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WinWinLay instead of R&B | Layout-to-image generation · COCO2014 | AP: 17.63 -> 19.74 | Helped |
| Use WinWinLay instead of AttRe | Layout-to-image generation · COCO2014 | FID: 27.51 -> 26.85 | Helped |
| Use WinWinLay instead of AttRe | Layout-to-image generation · COCO2014 | ImageReward: 0.7109 -> 0.7218 | Helped |
| Use WinWinLay instead of AttRe | Layout-to-image generation · Flickr30K | AP: 15.26 -> 17.28 | Helped |
| Use WinWinLay instead of AttRe | Layout-to-image generation · Flickr30K | FID: 27.72 -> 27.04 | Helped |
| Use WinWinLay instead of CSG | Layout-to-image generation · User Study | Controllability: 20.9 -> 27.7 | Helped |
| Use WinWinLay instead of AttRe | Layout-to-image generation · User Study | Quality: 22.3 -> 29.8 | Helped |
| Add non-local attention energy function to the original attention energy function + backpropagation update (ablation) | WinWinLay · COCO2014 | AP₅₀: 25.8 -> 44.1 | Helped |
| Add non-local attention energy function to the original attention energy function + backpropagation update (ablation) | WinWinLay · COCO2014 | AP: 8.4 -> 17.4 | Helped |
| Add adaptive update to the original attention energy function + backpropagation update (ablation) | WinWinLay · COCO2014 | CLIP-s: 0.310 -> 0.324 | Helped |
| Use both the non-local attention energy function and adaptive update instead of non-local attention energy function + backpropagation update (ablation) | WinWinLay · COCO2014 | AP: 17.4 -> 19.7 | Helped |
| Use both the non-local attention energy function and adaptive update instead of attention energy function + adaptive update (ablation) | WinWinLay · COCO2014 | CLIP-s: 0.324 -> 0.327 | Helped |
Surprising findings
- The paper theoretically shows that the standard attention energy function biases attention toward patches with larger initial values, so objects collapse into smaller regions of the box.
Limitations
- Adaptive update adds Langevin iterations, which increase generation time relative to vanilla backpropagation.
- Over-strong non-local prior can produce unnatural placements (e.g., a rabbit on a square canvas), requiring hyperparameter tuning of rho.
- Evaluation relies on a single base model (SD 1.5) and a self-selected dataset for qualitative comparison.
Experimental setup
Datasets: COCO2014 Flickr30K
Benchmarks: COCO2014 Flickr30K
Metrics: AP AP50 CLIP-s FID PickScore ImageReward user study controllability user study quality
Baselines: Layout-Control AttRe R&B CSG
Builds on: backward guidance Layout-Control Nash-MTL Langevin dynamics
Closest papers by meaning
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.89)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.89)
- SpotActor: Training-Free Layout-Controlled Consistent Image Generation (AAAI 2025 · similarity 0.88)
- Spatial Transport Optimization by Repositioning Attention Map for Training-Free Text-to-Image Synthesis (CVPR 2025 · similarity 0.88)
- SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation (AAAI 2024 · similarity 0.88)
- InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.