Research map / Image generation / Autoregressive and latent generation
Compose and Conquer: Diffusion-Based 3D Depth Aware Composable Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compose and Conquer (CnC) adds depth-disentangled local conditioning and mask-based soft guidance over CLIP image embeddings to Stable Diffusion, enabling 3D-aware object placement and region-specific global semantics.
Problem
Text-conditional diffusion models struggle to place multiple objects at correct depths in 3D and to apply global style/semantic conditions from multiple exemplar images to specific image regions without explicit localization cues.
Approach
Depth disentanglement training processes foreground and background depth maps from synthetic image triplets (generated by inpainting) in separate streams so the model learns relative depth ordering; soft guidance masks cross-attention between CLIP image tokens and latent regions using the foreground mask, so each exemplar semantic is applied only to its region.
Key results
- CnC Finetuned achieves FID 17.254, IS 32.131 and CLIP score 25.940 on COCO-Stuff semantics-only conditioning, best among compared models.
- On Pick-a-Pic depth+semantics, CnC Finetuned reaches FID 22.484 and CLIP score 28.094, beating Uni-ControlNet (22.808, 26.722).
- On reconstruction, CnC Finetuned gets LPIPS 0.6216 and MAE 0.1027 on Pick-a-Pic vs 0.6504 and 0.1111 for Uni-ControlNet; on COCO-Stuff depth-only FID is worse than ControlNet (22.257 vs 17.303).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the finetuning phase after independently training the local and global fusers | CnC · COCO-Stuff val, semantics | FID: 35.178 -> 17.254 | Helped |
| Add the finetuning phase after independently training the local and global fusers | CnC · COCO-Stuff val, depth | FID: 19.804 -> 22.257 | Hurt |
| Train with depth disentanglement training on synthetic foreground/background triplets rather than depth maps of the original image (ablation) | CnC local fuser · Relative-depth object placement | depth-aware placement | Helped |
| Add soft guidance to restrict foreground and background global-semantic tokens to their corresponding regions (ablation) | CnC global fuser · Conflicting localized semantics | semantic localization | Helped |
| Replace depth maps with canny edges as the local-fuser representation (ablation) | CnC · COCO-Stuff val | CLIPScore: +0.413 CLIPScore | Mixed |
| Increase the background semantic weight beyond the balanced setting (ablation) | CnC global fuser · Conflicting semantics | average cosine similarity | Mixed |
Surprising findings
- Finetuned CnC underperforms baselines on FID and IS for depth-only conditioning on both datasets.
- Canny-edge variant of the local fuser is competitive with baselines, and canny edges are preferred for some flat base images.
Limitations
- Limited number of available conditions; disentangled spatial grounding restricted to foreground and background only.
- Depth map inputs with inpainted background may not match the dataset prior, possibly hurting FID/IS for depth-only conditioning.
- Soft guidance discards the spatial information of the mask, so semantics are localized without explicit spatial cues.
Experimental setup
Datasets: COCO-Stuff Pick-a-Pic LAION
Benchmarks: COCO-Stuff val Pick-a-Pic val
Metrics: FID Inception Score CLIPScore SSIM LPIPS MAE of depth maps
Baselines: GLIGEN ControlNet Uni-ControlNet T2I-Adapter Composer
Builds on: Stable Diffusion Uni-ControlNet ControlNet LaMa
Closest papers by meaning
- $\textit{Bifr\"ost}$: 3D-Aware Image Compositing with Language Instructions (NeurIPS 2024 · similarity 0.88)
- Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformers (NeurIPS 2025 · similarity 0.88)
- Training-free Dense-Aligned Diffusion Guidance for Modular Conditional Image Synthesis (CVPR 2025 · similarity 0.87)
- Disentangled 3D Scene Generation with Layout Learning (ICML 2024 · similarity 0.87)
- CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-step (NeurIPS 2025 · similarity 0.87)
- Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.