Research map / Image generation / Text-to-image diffusion
SpotActor: Training-Free Layout-Controlled Consistent Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpotActor, a training-free pipeline for layout-to-consistent-image generation that jointly updates latent codes and semantic embeddings (dual energy guidance) on SDXL, plus a new benchmark ActorBench.
Problem
Text-to-image diffusion models cannot simultaneously place multiple subjects in given bounding boxes and keep each subject's appearance consistent across a set of images (e.g., comic production).
Approach
Treats latent and semantic (text embedding) spaces as one dual space, optimizing both with a sigmoid-like layout energy in a backward stage, then uses inter-image Regional Interconnection Self-Attention and Semantic Fusion Cross-Attention in the forward sampling stage to keep subjects consistent.
Key results
- SpotActor reaches 67.1% mIoU on ActorBench versus 53.7% for LayoutGuidance and 32.1% for SDXL.
- Subject consistency: 78.6 DINO-fg and 79.7 CLIP-fg, best among methods, versus 75.2 and 78.1 for StoryDiffusion.
- CLIP-T prompt conformity of 63.6, second only to SDXL's 65.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SpotActor instead of LayoutGuidance | SDXL · ActorBench | mIoU: 53.7 -> 67.1 | Helped |
| Use SpotActor instead of StoryDiffusion | SDXL · ActorBench | DINO-fg: 75.2 -> 78.6 | Helped |
| Use SpotActor instead of StoryDiffusion | SDXL · ActorBench | CLIP-fg: 78.1 -> 79.7 | Helped |
| Use SpotActor instead of SDXL | SDXL · ActorBench | CLIP-T: 65.4 -> 63.6 | Hurt |
| Add backward update (ablation) | SpotActor · ActorBench | mIoU: 30.8 -> 67.1 | Helped |
| Add RISA (ablation) | SpotActor · ActorBench | DINO-fg: 60.4 -> 78.6 | Helped |
| Add RISA (ablation) | SpotActor · ActorBench | CLIP-fg: 68.6 -> 79.7 | Helped |
| Add SFCA (ablation) | SpotActor · ActorBench | DINO-fg: 74.8 -> 78.6 | Helped |
| Add SFCA (ablation) | SpotActor · ActorBench | CLIP-fg: 72.2 -> 79.7 | Helped |
| Use sigmoid-like target function rather than binary mask or Gaussian (ablation) | SpotActor layout energy · Layout control | layout alignment | Helped |
Surprising findings
- Removing the backward update raises foreground consistency (DINO-fg 80.1) but drops mIoU to 30.8, showing a layout–consistency trade-off.
Limitations
- Performance depends on the base model's learned distribution, causing poor alignment for very small boxes, boxes near edges, or unusual layouts (OOD issues).
- Layout alignment and subject consistency trade off; removing backward update improves consistency but hurts mIoU.
Experimental setup
Datasets: COCO2017
Benchmarks: ActorBench
Metrics: mIoU DINO-fg CLIP-fg LPIPS-fg CLIP-T DINO-bg CLIP-bg LPIPS-bg
Baselines: SDXL StoryDiffusion LayoutGuidance
Builds on: LayoutGuidance StoryDiffusion energy guidance Stable Diffusion XL
Closest papers by meaning
- Control and Realism: Best of Both Worlds in Layout-to-Image without Training (ICML 2025 · similarity 0.88)
- Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation (NeurIPS 2024 · similarity 0.88)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.88)
- SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation (AAAI 2024 · similarity 0.88)
- InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention (NeurIPS 2025 · similarity 0.88)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.