Research map / 3D vision / 3D generation
OSN: Infinite Representations of Dynamic 3D Scenes from Monocular Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OSN, which learns a distribution of valid object scale ranges for monocular dynamic scenes so that many plausible 3D scene configurations can be sampled, rather than one most-plausible reconstruction.
Problem
Monocular dynamic 3D scene reconstruction is ill-posed: many 3D scene configurations (object scales) match the same video, yet existing methods recover only a single solution using extra priors.
Approach
An MLP object scale network predicts validity of joint normalized object scale combinations; per-object TensoRF representations are scale-invariant and trained jointly with the scale network via scaled composite rendering and soft Z-buffer rendering (pseudo validity labels from segmentation agreement), enabling infinite sampling of valid scene configurations.
Key results
- On the Dynamic Indoor Scene Dataset, OSN achieves PSNR 25.984 / SSIMAE 0.094 versus 24.695 PSNR and 0.137 SSIMAE for the best baseline Total-Recon (2)+3)).
- On the Gnome House 50-ground-truth benchmark, OSN reaches PSNR 22.940 ± 1.004 versus at most 18.846 for baselines under depth setting 3.
- Object-scale MSE averaged over four scenes is 0.064 for OSN versus 0.114 for the best baseline Total-Recon.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use OSN to model dynamic novel views | Dynamic Indoor Scene Dataset · Dynamic Indoor Scene Dataset (per-object SfM depth) | PSNR: 24.695 -> 25.984 | Helped |
| use OSN to model dynamic novel views | Oxford Multimotion Dataset · Oxford Multimotion Dataset (per-object SfM depth) | PSNR: 18.331 -> 19.671 | Helped |
| use OSN to model dynamic novel views | NVIDIA Dynamic Scene Dataset · NVIDIA Dynamic Scene Dataset | PSNR: 29.452 -> 29.588 | Helped |
| use OSN to match multiple plausible scale configurations | Gnome House · 50 Ground Truth Scenes of Gnome House (per-object SfM depth) | PSNR: 18.768±1.535 -> 22.940±1.004 | Helped |
| add early independent per-object bootstrapping (ablation) | OSN · Dynamic Indoor Scene Dataset | PSNR: 24.467 -> 25.984 | Helped |
| use 5 rather than 1 round of alternative optimization (ablation) | OSN · Dynamic Indoor Scene Dataset | PSNR: 23.123 -> 25.984 | Helped |
| use 10 rather than 5 rounds of alternative optimization (ablation) | OSN · Dynamic Indoor Scene Dataset | PSNR: 25.984 -> 25.065 | Hurt |
| use 10 rather than 5 rounds of alternative optimization (ablation) | OSN · Dynamic Indoor Scene Dataset | SSIMAE: 0.094 -> 0.108 | Hurt |
Surprising findings
- Adding MiDaS depth priors does not consistently improve baselines and can introduce unreliable geometry and temporal inconsistency.
Limitations
- Restricted to rigid objects; deformable scenes left for future work.
- Performance on KITTI is only competitive because forward-moving camera motion causes SfM and tracking failures.
- Object scale network is a small 4-layer MLP, and the paper notes it may need more layers for many objects.
Experimental setup
Datasets: Dynamic Indoor Scene Dataset (adapted from NVFi) Oxford Multimotion Dataset NVIDIA Dynamic Scene Dataset KITTI Sequence 0007 Chessboard++ (synthetic)
Benchmarks: Dynamic Indoor Scene Dataset Oxford Multimotion Dataset NVIDIA Dynamic Scene Dataset Gnome House 50-ground-truth benchmark KITTI Sequence 0007 Chessboard++
Metrics: PSNR SSIM LPIPS SSIMAE PQ mIoU MSE of object scales
Baselines: NSFF DynNeRF TiNeuVox HexPlane Total-Recon
Builds on: TensoRF Total-Recon SAM RAFT NSFF
Closest papers by meaning
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.89)
- DyST: Towards Dynamic Neural Scene Representations on Real-World Videos (ICLR 2024 · similarity 0.88)
- Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos (NeurIPS 2025 · similarity 0.88)
- From an Image to a Scene: Learning to Imagine the World from a Million 360° Videos (NeurIPS 2024 · similarity 0.88)
- World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.88)
- DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.