Research map / 3D vision / Gaussian splatting
Robust Multi-Object 4D Generation for In-the-wild Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GenMOJO, which jointly splats deformable 3D Gaussians of multiple objects from monocular videos and optimizes them with object-centric diffusion SDS priors via affine transformations, and introduces the MOSE-PTS point-track benchmark.
Problem
Generating complete, temporally coherent 4D scenes from monocular multi-object videos with heavy occlusions, where existing video-to-4D methods struggle with inter-object occlusion and object-centric generative priors do not directly apply to global scenes.
Approach
Compose all object Gaussians early in a global frame with depth initialization and jointly optimize their deformations via occlusion-aware multi-object splatting, instance-mask rendering loss, and differentiable 2D affine transformations that let object-centric SDS from Zero-1-to-3 supervise each object in its own canonical frame.
Key results
- On MOSE, GenMOJO reaches PSNR 25.56 vs 22.98 for DreamScene4D and CLIP 85.41 vs 85.16.
- In human two-way voting, GenMOJO is preferred 63.2% over DreamScene4D.
- On MOSE point tracking, GenMOJO achieves ATE 9.91 vs 13.47 for DreamScene4D and 18.39 for Shape of Motion, though supervised CoTrackerv3 gets 11.54.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GenMOJO instead of DreamScene4D | 4D scene generation · MOSE | PSNR: 22.98 -> 25.56 | Helped |
| use GenMOJO instead of DreamScene4D | 4D scene generation · DAVIS | PSNR: 21.73 -> 23.51 | Helped |
| use GenMOJO instead of DreamScene4D | point tracking · MOSE | A-EPE: 22.48 -> 15.46 | Helped |
| use GenMOJO instead of DreamScene4D | point tracking · DAVIS | A-EPE: 17.47 -> 14.14 | Helped |
| add multi-object joint splatting (ablation) | GenMOJO · MOSE | A-EPE: 23.48 -> 15.46 | Helped |
| add transformed object-centric rendering (ablation) | GenMOJO · MOSE | A-EPE: 19.44 -> 15.46 | Helped |
| add instance mask rendering (ablation) | GenMOJO · MOSE | A-EPE: 16.77 -> 15.46 | Helped |
| use object-centric SDS instead of scene-level SDS (ablation) | GenMOJO · MOSE | A-EPE: 20.03 -> 15.46 | Helped |
| use DepthCrafter instead of DepthAnything-V2 (ablation) | GenMOJO · MOSE | A-EPE: 17.32 -> 15.46 | Helped |
Surprising findings
- Supervised point trackers CoTrackerv2/v3 perform markedly worse on MOSE than on DAVIS, showing MOSE's occlusion difficulty.
- Scene-level SDS (Zero-1-to-3 scene variant) lowered performance relative to object-centric SDS, suggesting object-centric NVS priors give better SDS gradients.
Limitations
- Relies on generalization of view-conditioned generative models, which struggle with unconventional camera poses.
- Jittery depth estimates can cause objects to jump along the depth axis.
- Test-time optimization makes the method unsuitable for online applications.
Experimental setup
Datasets: DAVIS MOSE
Benchmarks: MOSE-PTS DAVIS MOSE
Metrics: CLIP similarity LPIPS PSNR user preference ATE MTE A-EPE M-EPE
Baselines: Consistent4D DreamGaussian4D DreamScene4D Shape of Motion GMRW CoTrackerv2 CoTrackerv3
Builds on: DreamScene4D DreamGaussian4D Zero-1-to-3 Shape of Motion DepthCrafter
Closest papers by meaning
- DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos (NeurIPS 2024 · similarity 0.92)
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.89)
- L4GM: Large 4D Gaussian Reconstruction Model (NeurIPS 2024 · similarity 0.88)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.88)
- SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency (ICLR 2025 · similarity 0.88)
- 4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.