Research map / 3D vision / 3D generation
DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DreamScene4D, which decomposes monocular multi-object videos into background and object tracks, factorizes object motion into deformation, object-to-world transform, and camera motion, and lifts each object with SDS-based Gaussian splatting to compose a 4D scene.
Problem
Generating complete 3D/4D multi-object dynamic scenes with large object motion from monocular videos, where rendering gradients are weak for fast motion and view-predictive diffusion priors work poorly at scene level.
Approach
A decompose-recompose strategy: segment and amodally complete each object track, lift each object in an object-centric frame with SDS and Zero-1-to-3, factorize motion into object-centric deformation (K-plane/Hexplane), object-to-world affine transforms from bounding boxes, and camera motion, then recompose objects using monocular depth guidance.
Key results
- On DAVIS, DreamScene4D reaches CLIP 85.09 and LPIPS 0.152 versus 82.14 / 0.141 for Consistent4D and 77.81 / 0.181 for DreamGaussian4D.
- On Kubric, mean EPE drops to 14.30 for DreamScene4D versus 101.79 for DreamGaussian4D, and 2.51 for the supervised CoTracker.
- In a user study on 30 DAVIS videos, DreamScene4D was preferred over DreamGaussian4D 77.9% to 22.1% and over Consistent4D 71.7% to 28.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Video Scene Decomposition to DreamGaussian4D | DreamGaussian4D · DAVIS | EPE (vis): 26.65 -> 20.95 | Helped |
| Add motion factorization to DreamGaussian4D with Video Scene Decomposition | DreamGaussian4D w/ VSD · DAVIS | EPE (vis): 20.95 -> 8.56 | Helped |
| Add motion factorization to DreamGaussian4D with Video Scene Decomposition | DreamGaussian4D w/ VSD · Kubric | Mean EPE: 85.27 -> 14.30 | Helped |
| Remove flow loss (ablation) | DreamScene4D · DAVIS | EPE (vis): 8.56 -> 10.91 | Hurt |
| Remove flow loss (ablation) | DreamScene4D · DAVIS | EPE (occ): 4.24 -> 3.83 | Helped |
| Remove flow loss (ablation) | DreamScene4D · Kubric | Mean EPE: 14.30 -> 18.54 | Hurt |
| Remove rigid and scale losses (ablation) | DreamScene4D · DAVIS | EPE (vis): 8.56 -> 10.29 | Hurt |
| Remove rigid and scale losses (ablation) | DreamScene4D · Kubric | Mean EPE: 14.30 -> 16.21 | Hurt |
| Remove flow loss (ablation) | DreamScene4D · DAVIS | CLIP: 85.09 -> 84.94 | No clear effect |
| Remove flow loss (ablation) | DreamScene4D · Kubric | CLIP: 85.53 -> 86.41 | Helped |
| Remove rigid and scale losses (ablation) | DreamScene4D · DAVIS | CLIP: 85.09 -> 83.24 | Hurt |
Surprising findings
- DreamScene4D's motion EPE on DAVIS and Kubric is reported to beat PIPS++, a point tracker trained on tracking data, despite never being trained on point tracking data.
- Removing the flow loss gives slightly better Kubric CLIP and EPE on occluded points, so the flow term does not uniformly help.
Limitations
- SDS prior fails on videos captured at steep camera elevation angles.
- Heavy occlusions lead to under-constrained Gaussians and inpainting artifacts.
- Runtime scales linearly with number of objects and is slow for complex videos.
Experimental setup
Datasets: DAVIS Kubric TAP-Vid-DAVIS YoutubeVOS
Benchmarks: DAVIS Kubric TAP-Vid-DAVIS
Metrics: CLIP similarity LPIPS user preference endpoint error (EPE)
Baselines: Consistent4D DreamGaussian4D DreamGaussian4D w/ VSD PIPS++ CoTracker
Builds on: DreamGaussian DreamGaussian4D Zero-1-to-3 Score Distillation Sampling SD-Inpaint
Closest papers by meaning
- Robust Multi-Object 4D Generation for In-the-wild Videos (CVPR 2025 · similarity 0.92)
- SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency (ICLR 2025 · similarity 0.90)
- 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video (ICLR 2024 · similarity 0.90)
- GenXD: Generating Any 3D and 4D Scenes (ICLR 2025 · similarity 0.90)
- EG4D: Explicit Generation of 4D Object without Score Distillation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.