Research map / Image generation / Editing and style transfer
Move Anything with Layered Scene Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free SceneDiffusion optimizes a layered scene representation during diffusion sampling by jointly denoising multiple randomly sampled layouts, enabling moving, resizing, cloning and restyling of objects with general T2I diffusion models.
Problem
Controllable scene generation and spatial editing with text-to-image diffusion models, which lack spatially disentangled mid-level representations because of their fixed forward noising process.
Approach
Optimize a layered scene representation (per-layer feature maps with masks and offsets) during diffusion sampling by denoising several randomly sampled layouts in parallel and solving a closed-form least-squares update for the feature maps; a short image diffusion stage then renders the scene, and a reference-image trajectory anchors edits to in-the-wild images.
Key results
- Controllable scene generation: Mask IoU 0.522 and Consistency 0.721 vs. MultiDiffusion's 0.466 and 0.436.
- Object moving: FID 5.289 and Mask IoU 0.817 vs. best inpainting baseline FID 6.383 and Mask IoU 0.747.
- Rendering a new layout takes about 0.82s on a single GPU with N=8, tau=13 (optimization ~17.3s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SceneDiffusion instead of MultiDiffusion | controllable scene generation · Controllable scene generation | Mask IoU: 0.466 -> 0.522 | Helped |
| use SceneDiffusion instead of MultiDiffusion | controllable scene generation · Controllable scene generation | Consistency: 0.436 -> 0.721 | Helped |
| use SceneDiffusion instead of MultiDiffusion | controllable scene generation · Controllable scene generation | LPIPS: 0.519 -> 0.215 | Helped |
| use SceneDiffusion instead of MultiDiffusion | controllable scene generation · Controllable scene generation | SSIM: 0.471 -> 0.762 | Helped |
| use SceneDiffusion instead of the inpainting baseline | object moving · Object moving | FID: 6.383 -> 5.289 | Helped |
| use SceneDiffusion instead of the inpainting baseline | object moving · Object moving | Mask IoU: 0.747 -> 0.817 | Helped |
| use SceneDiffusion instead of the inpainting baseline | object moving · Object moving | SSIM: 0.680 -> 0.709 | Helped |
| remove joint denoising of multiple layouts (ablation) | SceneDiffusion · Controllable scene generation | Consistency: 0.72 -> 0.43 | Hurt |
| use random rather than deterministic layout sampling (ablation) | SceneDiffusion · Controllable scene generation | CLIP-a: 5.98 -> 6.12 | Helped |
| remove the image diffusion rendering stage (ablation) | SceneDiffusion · Controllable scene generation | CLIP-a: 6.12 -> 5.96 | Hurt |
| remove the image diffusion rendering stage (ablation) | SceneDiffusion · Controllable scene generation | Visual Consistency: 0.11 -> 0.09 | Helped |
| reduce layout views from N=8 to N=4 (ablation) | SceneDiffusion · Controllable scene generation | Consistency: 0.721 -> 0.689 | Hurt |
Surprising findings
- Stopping SceneDiffusion earlier and rendering more steps with image diffusion trades consistency for quality; skipping the image diffusion stage slightly improves consistency metrics but significantly degrades image quality.
Limitations
- Object appearance may not fit tightly to the mask in the final rendered image.
- Simultaneous denoising of multiple layouts requires large memory, limiting use in resource-limited settings.
- Quantitative evaluation restricted to single-object scenes due to lack of multi-object spatial editing pairs.
Experimental setup
Datasets: curated subject-centric images generated from 1,000 captions (5,092 after filtering)
Benchmarks: curated 5,092-image single-object dataset
Metrics: Mask IoU Consistency Visual Consistency LPIPS SSIM FID CLIP-aesthetic
Baselines: MultiDiffusion RePaint Inpainting model (specialized, trained with masking)
Builds on: MultiDiffusion locally conditioned diffusion BlobGAN RePaint
Closest papers by meaning
- DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis (CVPR 2024 · similarity 0.90)
- A Diffusion-Based Framework for Occluded Object Movement (AAAI 2025 · similarity 0.89)
- DesignEdit: Unify Spatial-Aware Image Editing via Training-free Inpainting with a Multi-Layered Latent Diffusion Framework (AAAI 2025 · similarity 0.89)
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.89)
- CTRL-D: Controllable Dynamic 3D Scene Editing with Personalized 2D Diffusion (CVPR 2025 · similarity 0.88)
- PS-Diffusion: Photorealistic Subject-Driven Image Editing with Disentangled Control and Attention (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.