Research map / 3D vision / 3D generation
Coherent 3D Scene Diffusion From a Single RGB Image
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an image-conditioned 3D scene diffusion model that jointly denoises 7-DoF object poses and Gaussian-based shape codes for all objects in a scene from a single RGB image, with a surface alignment loss enabling training under partial supervision.
Problem
Single-RGB-image 3D scene reconstruction is ill-posed and existing methods reconstruct objects independently, producing intersecting or floating objects and noisy shapes; public datasets also provide only partial ground-truth annotations.
Approach
Model the joint distribution of all objects in a scene with a conditional diffusion process, using intra-scene attention so objects exchange relational information during denoising, and supervise shape-pose alignment via a one-sided Chamfer loss on points sampled directly from the intermediate Gaussian shape representation.
Key results
- On SUN RGB-D, mAP^15_3D of 58.18 vs 46.14 for Im3D (+12.04 points), with IoU_3D 36.10 vs 28.31.
- On Pix3D, F-Score 58.71 vs 45.28 for InstPIFu (+13.43) and Chamfer distance 15.05 vs 24.65 for InstPIFu.
- Ablation: removing diffusion (1-step regression) drops mAP^15_3D by 11.08 points; removing joint L_align training drops it by 2.11 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed method instead of Im3D | 3D scene reconstruction · SUN RGB-D | AP^15_3D: 46.14 -> 58.18 | Helped |
| use the proposed method instead of Im3D | 3D scene reconstruction · SUN RGB-D | L_align: 1.24 -> 0.99 | Helped |
| use the proposed method instead of InstPIFu | 3D shape reconstruction · Pix3D | CD × 10^3: 24.65 -> 15.05 | Helped |
| use the proposed method instead of InstPIFu | 3D shape reconstruction · Pix3D | F-Score: 45.28 -> 58.71 | Helped |
| use denoising diffusion instead of one-step regression (ablation) | Ours · SUN RGB-D | AP^15_3D: 47.10 -> 58.18 | Helped |
| use denoising diffusion instead of one-step regression (ablation) | Ours · SUN RGB-D | L_align: 1.18 -> 0.99 | Helped |
| add intra-scene attention (ablation) | Ours · SUN RGB-D | AP^15_3D: 48.88 -> 58.18 | Helped |
| add intra-scene attention (ablation) | Ours · SUN RGB-D | L_align: 1.12 -> 0.99 | Helped |
| add joint pose-shape training (ablation) | Ours · SUN RGB-D | AP^15_3D: 56.07 -> 58.18 | Helped |
| add joint pose-shape training (ablation) | Ours · SUN RGB-D | L_align: 1.06 -> 0.99 | Helped |
| add image-conditioning using Mask2Former detections (ablation) | Ours · SUN RGB-D | mAP^15_3D: 24.17 -> 33.58 | Helped |
| add the scene prior and inter-object relationships (ablation) | Ours · SUN RGB-D | mAP^15_3D: +7.11% mAP | Helped |
Surprising findings
- Im3D outperforms the proposed method on the 'bed' class in SUN RGB-D, attributed to frequent partial visibility of beds.
Limitations
- Depends on accurate 2D object detection and instance segmentation (oracle 2D detections used in evaluation).
- Shape prior does not generalize to unseen object categories.
- Assumes static scene geometry; no affordance or articulation modeling.
Experimental setup
Datasets: SUN RGB-D Pix3D ScanNet COCO ShapeNet Scan2CAD
Benchmarks: SUN RGB-D Pix3D ScanNet
Metrics: IoU_3D AP^15_3D L_align Chamfer distance F-Score alignment accuracy
Baselines: Total3D Im3D InstPIFu ROCA SPAGHETTI
Builds on: DDPM Mask2Former SPAGHETTI Total3D Im3D
Closest papers by meaning
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.91)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.90)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.89)
- Wonderland: Navigating 3D Scenes from a Single Image (CVPR 2025 · similarity 0.89)
- DiffusionSfM: Predicting Structure and Motion via Ray Origin and Endpoint Diffusion (CVPR 2025 · similarity 0.89)
- DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.