Research map / 3D vision / 3D generation
DORSal: Diffusion for Object-centric Representations of Scenes $\textit{et al.}$
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DORSal, which conditions a video diffusion U-Net on frozen OSRT object slots to render sharp novel views of multi-object and Street View scenes, enabling object removal and transfer edits.
Problem
Scene-representation-based novel-view synthesis from few views produces blurry renders because of L2 training under uncertainty, while diffusion NVS models struggle to scale to complex multi-object scenes and to support object-level editing.
Approach
Condition a video diffusion (Multiview U-Net with axial attention) on frozen object-centric OSRT slots and target camera poses via cross-attention and FiLM, so the generative decoder adds sharp details while object slots enable slot-dropout-based object removal and slot transfer between scenes.
Key results
- On MultiShapeNet, DORSal achieves FID 11.01 versus 67.29 for SRT and 100.7 for OSRT, though its PSNR (18.76) is lower than SRT (25.93).
- Under matched DDPM sampling, DORSal beats 3DiM on Street View with PSNR 16.36 vs 12.68 and FID 14.62 vs 15.58.
- Object removal via slot dropout on MultiShapeNet reaches 45.8 FG-mIoU for DORSal versus 43.1 for OSRT, with lower FG-ARI (70.0 vs 79.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Condition novel-view synthesis on object slots and use a diffusion decoder instead of OSRT rendering | DORSal vs. OSRT · MultiShapeNet | FID: 100.7 -> 11.01 | Helped |
| Condition novel-view synthesis on object slots and use a diffusion decoder instead of OSRT rendering | DORSal vs. OSRT · Street View | FID: 165.1 -> 16.24 | Helped |
| Use DORSal instead of SRT | Novel-view synthesis · MultiShapeNet | PSNR: 25.93 -> 18.76 | Hurt |
| Use DORSal instead of 3DiM with DDPM sampling | Novel-view synthesis · Street View | PSNR: 12.68 -> 16.36 | Helped |
| Use DORSal instead of 3DiM with DDPM sampling | Novel-view synthesis · MultiShapeNet | FID: 10.94 -> 9.00 | Helped |
| Use 64x64 rather than 128x128 resolution (ablation) | DORSal scene editing · MultiShapeNet | FG-mIoU: 60.6 -> 70 | Helped |
| Add slot dropout to the conditioning slots (ablation) | DORSal with Sup-OSRT · MultiShapeNet | FG-ARI | Hurt |
| Add slot dropout to the conditioning slots (ablation) | DORSal with OSRT · Street View | scene editing quality | Helped |
| Remove instance segmentation supervision from the OSRT base model (ablation) | DORSal · MultiShapeNet | scene editing performance | Hurt |
| Increase inference guidance weight above 1, with an optimum at 2 (ablation) | DORSal · Novel-view synthesis | prediction quality | Helped |
| Smooth edit differences with a median filter up to kernel size 7 (ablation) | DORSal (128x128) scene-edit evaluation · MultiShapeNet | segmentation score | Mixed |
| Train on nearby and randomly sampled views rather than randomly sampled views alone (ablation) | DORSal camera-path rendering · MultiShapeNet camera paths | PSNR: 16.50db -> 18.06db | Helped |
Surprising findings
- DORSal obtains a higher FG-mIoU than the unsupervised OSRT base model on MultiShapeNet despite no instance-level architectural priors or segmentation supervision.
Limitations
- DORSal is not trained end-to-end and is limited by the quality of the frozen OSRT object slots.
- Higher-resolution (128x128) model shows worse editing performance and cross-view consistency than the 64x64 model.
- Object edits have non-local side effects, and many Street View slots have little or no effect when removed.
Experimental setup
Datasets: MultiShapeNet (MSN-Hard) Street View
Benchmarks: MultiShapeNet (MSN-Hard) Street View
Metrics: PSNR LPIPS FID FG-mIoU FG-ARI
Baselines: SRT OSRT Sup-OSRT 3DiM
Builds on: Object Scene Representation Transformer (OSRT) Video Diffusion Models 3DiM Slot Attention Denoising Diffusion Probabilistic Models
Closest papers by meaning
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.89)
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.89)
- Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.89)
- Taming Video Diffusion Prior with Scene-Grounding Guidance for 3D Gaussian Splatting from Sparse Inputs (CVPR 2025 · similarity 0.88)
- Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.