Research map / Image generation / Text-to-image diffusion
Zero-Painter: Training-Free Layout Control for Text-to-Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Zero-Painter is a training-free two-stage layout-to-image method that generates each masked object with prompt-adjusted cross-attention and then composes them via region-grouped cross-attention in a Stable Inpainting model.
Problem
Layout-conditional text-to-image synthesis, where generated objects often fail to match their mask shapes and per-object attributes leak between objects in multi-object prompts.
Approach
Generate each object separately on a flat background using a mask-initialized starting latent and a modified cross-attention (PACA) that boosts mask-region similarity to object prompt tokens, then segment with SAM and compose the objects with an inpainting model whose cross-attention is grouped by region (ReGCA), all without training.
Key results
- Zero-Painter achieves local IoU 0.75 vs 0.58 for MultiDiffusion, 0.50 for NTLB and 0.62 for eDiff-I on 3000 MSCOCO layouts.
- Local CLIP score of 26.68 for Zero-Painter, the highest among compared methods (MultiDiffusion 26.10, GLIGEN 25.52, eDiff-I 25.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Zero-Painter instead of MDF | Layout-conditional text-to-image generation · MSCOCO validation layouts | CLIP (local): 26.10 -> 26.68 | Helped |
| Use Zero-Painter instead of EDIFF-I | Layout-conditional text-to-image generation · MSCOCO validation layouts | IoU (local): 0.62 -> 0.75 | Helped |
| Add PACA to single-object generation (ablation) | Zero-Painter · Single-object generation | shape alignment | Helped |
| Add PACA to single-object generation (ablation) | Zero-Painter · Single-object generation | textual fidelity | Helped |
| Add ReGCA instead of using a basic inpainting pipeline (ablation) | Zero-Painter · Comprehensive composition | object shape preservation | Helped |
| Add ReGCA instead of using a basic inpainting pipeline (ablation) | Zero-Painter · Comprehensive composition | background inpainting quality | Helped |
Limitations
- Overlapping masks produce unnatural or less coherent outputs because objects inside masks are not altered during composition.
- Quality can degrade from the approximation of the starting latent, mitigated only by a refinement sub-stage.
Experimental setup
Datasets: MSCOCO segmentation layouts (3000 validation layouts)
Benchmarks: MSCOCO validation layouts (3000, filtered)
Metrics: Local CLIP score Local IoU (SAM-based)
Baselines: eDiff-I GLIGEN NTLB MultiDiffusion
Builds on: Stable Diffusion eDiff-I Paint-With-Words MultiDiffusion Stable Inpainting Segment Anything
Closest papers by meaning
- Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator (CVPR 2025 · similarity 0.87)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.87)
- PreciseCam: Precise Camera Control for Text-to-Image Generation (CVPR 2025 · similarity 0.87)
- Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models (AAAI 2024 · similarity 0.87)
- Devil is in the Detail: Towards Injecting Fine Details of Image Prompt in Image Generation via Conflict-free Guidance and Stratified Attention (CVPR 2025 · similarity 0.87)
- UNIC-Adapter: Unified Image-instruction Adapter with Multi-modal Transformer for Image Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.