Research map / Image generation / Text-to-image diffusion
NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NoiseCollage, a training-free layout-aware text-to-image diffusion method that estimates per-object noises with masked cross-attention and crop-and-merges them into one denoising noise, also integrated with ControlNet.
Problem
Layout-aware text-to-image generation for multiple objects suffers from mismatches between text and layout conditions (attention manipulation) and image quality degradation over iterations (iterative editing).
Approach
Independently estimate noise for each object and the global image, then crop each object noise by its layout region and weighted-merge them with the global noise, using masked cross-attention so each object's text only affects its region.
Key results
- On MD30, NoiseCollage achieves average CLIP text-region similarity 0.280 vs 0.253 for Collage Diffusion and 0.250 for Paint-with-words.
- On BD807, NoiseCollage scores 0.256 vs 0.240 (Paint-with-words) and 0.237 (Collage Diffusion).
- Integrating with ControlNet raises pose-condition similarity on HMD20 from 0.297 to 0.332 and on HBD256 from 0.250 to 0.280.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NoiseCollage instead of Paint-with-words | Layout-aware text-to-image generation · MD30 | CLIP similarity: 0.250 -> 0.280 | Helped |
| Use NoiseCollage instead of Collage Diffusion | Layout-aware text-to-image generation · MD30 | CLIP similarity: 0.253 -> 0.280 | Helped |
| Use NoiseCollage instead of Paint-with-words | Layout-aware text-to-image generation · BD807 | CLIP similarity: 0.240 -> 0.256 | Helped |
| Use NoiseCollage instead of Collage Diffusion | Layout-aware text-to-image generation · BD807 | CLIP similarity: 0.237 -> 0.256 | Helped |
| Apply NoiseCollage to ControlNet | ControlNet · RMD30 (edge) | CLIP similarity: 0.293 -> 0.307 | Helped |
| Apply NoiseCollage to ControlNet | ControlNet · RMD807 (edge) | CLIP similarity: 0.265 -> 0.284 | Helped |
| Apply NoiseCollage to ControlNet | ControlNet · RMD30 (sketch) | CLIP similarity: 0.276 -> 0.300 | Helped |
| Apply NoiseCollage to ControlNet | ControlNet · HMD20 (pose) | CLIP similarity: 0.297 -> 0.332 | Helped |
| Apply NoiseCollage to ControlNet | ControlNet · HBD256 (pose) | CLIP similarity: 0.250 -> 0.280 | Helped |
Surprising findings
- Small objects are often ignored by NoiseCollage, and Collage Diffusion shares this failure, while Paint-with-words shows them in wrong places and styles.
Limitations
- Sometimes ignores small objects in generated images.
- Layout must be given as bounding boxes or polygons; no automatic layout inference.
- Evaluation relies on CLIP similarity and small datasets (MD30 has only 30 images).
Experimental setup
Datasets: MS-COCO test set BD807 MD30 HBD256 HMD20
Benchmarks: BD807 MD30 HBD256 HMD20
Metrics: CLIP cosine similarity between region image feature and text sample-level win rate
Baselines: Paint-with-words Collage Diffusion ControlNet
Builds on: Stable Diffusion ControlNet classifier-free guidance UniPC
Closest papers by meaning
- InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization (CVPR 2024 · similarity 0.87)
- Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusion (CVPR 2025 · similarity 0.86)
- RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.86)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.86)
- InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention (NeurIPS 2025 · similarity 0.86)
- Check Locate Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.