Research map / Image generation / Text-to-image diffusion
R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes R&B, a training-free backward cross-attention guidance method for zero-shot layout-grounded text-to-image generation with Stable Diffusion, using dynamic thresholding, box selection via straight-through estimator, and region- and boundary-aware losses.
Problem
Text-to-image diffusion models often ignore or misplace objects relative to a specified layout (bounding boxes), and existing zero-shot layout guidance methods give imprecise spatial control and suffer semantic errors such as missing objects and attribute misbinding.
Approach
Binarize aggregated cross-attention maps with a dynamic threshold, take their minimum bounding rectangles, and connect these hard boxes to the soft attention maps via a straight-through estimator so a region-aware IoU-style loss can directly push boxes toward ground truth; add a Sobel-based boundary-aware loss to sharpen attention edges inside boxes, applied only in the first 10 denoising steps.
Key results
- On HRS spatial track, R&B reaches 30.14% vs 24.45% for Attention-refocusing and 8.48% for Stable Diffusion.
- On DrawBench spatial accuracy, R&B reaches 55.00% vs 43.50% for Attention-refocusing.
- On MS-COCO subset, R&B achieves mIoU 0.5533 vs 0.4460 for Layout-guidance and CLIP T2I-Sim 0.3218 vs 0.3108.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add region-aware loss (ablation) | R&B · HRS Spatial | accuracy (%): 20.01 -> 30.14 | Helped |
| add region-aware loss (ablation) | R&B · HRS Size | accuracy (%): 15.05 -> 26.74 | Helped |
| add region-aware loss (ablation) | R&B · HRS Color | accuracy (%): 23.22 -> 32.04 | Helped |
| add region-aware loss (ablation) | R&B · DrawBench Spatial | accuracy (%): 42.50 -> 55.00 | Helped |
| add boundary-aware loss (ablation) | R&B · HRS Spatial | accuracy (%): 25.99 -> 30.14 | Helped |
| add boundary-aware loss (ablation) | R&B · HRS Size | accuracy (%): 23.26 -> 26.74 | Helped |
| add boundary-aware loss (ablation) | R&B · HRS Color | accuracy (%): 27.32 -> 32.04 | Helped |
| add boundary-aware loss (ablation) | R&B · DrawBench Spatial | accuracy (%): 47.50 -> 55.00 | Helped |
| use discrete sampling (ablation) | R&B · MS-COCO subset | mIoU: 0.5106 -> 0.5533 | Helped |
| use discrete sampling (ablation) | R&B · MS-COCO subset | T2I-Sim: 0.3142 -> 0.3218 | Helped |
| set guidance ratio to 300 rather than 70 (ablation) | R&B · MS-COCO | mIoU: 0.5533 -> 0.4818 | Hurt |
| set guidance ratio to 300 rather than 70 (ablation) | R&B · MS-COCO | T2I-Sim: 0.3218 -> 0.3007 | Hurt |
Surprising findings
- Attention-refocusing scores high on HRS/DrawBench accuracy but its images often ignore box constraints, showing the benchmarks measure coarse aspects only.
- Optimizing only the sigmoid-sharpened soft mask produces artifacts; the linear normalized attention map is needed as a complement.
Limitations
- Fails on out-of-domain or uncommon prompts and unreasonable layouts because the base Stable Diffusion lacks coverage.
- Prompts with four or more objects lead to missing objects.
- Guidance strength eta_g must be tuned; excessive values degrade fidelity.
Experimental setup
Datasets: HRS DrawBench MS-COCO
Benchmarks: HRS DrawBench MS-COCO
Metrics: accuracy mean IoU CLIP-Score
Baselines: Stable Diffusion BoxDiff Layout-guidance Attention-refocusing GLIGEN
Builds on: Layout-guidance BoxDiff Attention-refocusing classifier guidance
Closest papers by meaning
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.91)
- Precise Parameter Localization for Textual Generation in Diffusion Models (ICLR 2025 · similarity 0.90)
- SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation (AAAI 2024 · similarity 0.90)
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.90)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.90)
- Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.