Research map / Image generation / Text-to-image diffusion
Check Locate Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SimM, a training-free check-locate-rectify system that edits Stable Diffusion cross-attention maps at inference to correct layout errors, and introduces the SimMBench benchmark of superlative spatial prompts.
Problem
Text-to-image diffusion models often fail to follow layout instructions in prompts, especially superlative spatial relations (e.g., "the crown on the bottom"), and existing layout-control methods need training or costly gradient-based guidance.
Approach
Parse the prompt with spaCy heuristics to get target boxes, check the first-step merged cross-attention for misplaced objects, locate the active region over early steps, and rectify by transferring activations and enhancing/suppressing intra- and inter-map attention, with no gradient updates.
Key results
- On DrawBench subset, SimM reaches 53.00% generation accuracy versus 43.50% for Attention-Refocusing and 12.50% for Stable Diffusion.
- On SimMBench (203 prompts), SimM reaches 65.16% accuracy, versus 50.71% for Attention-Refocusing.
- SimM achieves the best CLIP-Score on DrawBench (0.3423) but comparable-to-lower CLIP-Score on SimMBench (0.3001 vs 0.3032 for BoxDiff).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SimM instead of Attention-Refocusing | Stable Diffusion · DrawBench | accuracy (%): 43.50 -> 53.00; 9.5% | Helped |
| Apply SimM instead of Attention-Refocusing | Stable Diffusion · SimMBench | accuracy (%): 50.71 -> 65.16; 14.45% | Helped |
| Apply SimM instead of Layout-Guidance | Stable Diffusion · DrawBench | CLIP-Score: 0.3354 -> 0.3423 | Helped |
| Apply SimM instead of BoxDiff | Stable Diffusion · SimMBench | CLIP-Score: 0.3032 -> 0.3001 | Hurt |
| Add intra-map activation enhancement and suppression (ablation) | SimM · Text-to-image layout generation | object placement and presence | Helped |
| Add inter-map activation enhancement and suppression (ablation) | SimM · Text-to-image layout generation | object generation correctness | Helped |
| Start rectification earlier by using T loc = 1 (ablation) | SimM · Text-to-image layout generation | layout fidelity | Helped |
| Start rectification later by increasing T loc (ablation) | SimM · Text-to-image layout generation | layout fidelity | Hurt |
| Set adjustment strength α to 0.1 (ablation) | SimM · Text-to-image layout generation | layout control | Hurt |
| Set adjustment strength α to 1 (ablation) | SimM · Text-to-image layout generation | layout rectification | Hurt |
| Set adjustment strength α to 10 (ablation) | SimM · Text-to-image layout generation | layout rectification | Helped |
| Increase adjustment strength α to 50 (ablation) | SimM · Text-to-image layout generation | image quality | Hurt |
Surprising findings
- CLIP-Score is reported as unreliable for spatial relations, and SimM's CLIP-Score on SimMBench is slightly below several baselines.
Limitations
- Relies on heuristic vocabulary and dependency-parsing rules to derive target boxes, which may fail on complex prompts.
- Benchmark is small (203 prompts) and evaluation uses only four images per prompt.
- CLIP-Score does not capture spatial correctness, so it is a weak signal for layout fidelity.
Experimental setup
Datasets: DrawBench SimMBench
Benchmarks: DrawBench SimMBench
Metrics: generation accuracy CLIP-Score
Baselines: Stable Diffusion BoxDiff Layout-Guidance Attention-Refocusing
Builds on: Stable Diffusion Layout-Guidance Attention-Refocusing BoxDiff DDIM
Closest papers by meaning
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.89)
- Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation (NeurIPS 2024 · similarity 0.87)
- SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation (AAAI 2024 · similarity 0.87)
- Control and Realism: Best of Both Worlds in Layout-to-Image without Training (ICML 2025 · similarity 0.87)
- SpotActor: Training-Free Layout-Controlled Consistent Image Generation (AAAI 2025 · similarity 0.87)
- Layout-Agnostic Scene Text Image Synthesis with Diffusion Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.