Research map / 3D vision / 3D generation
From an Image to a Scene: Learning to Imagine the World from a Million 360° Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces 360-1M, a million-video 360° YouTube dataset with a scalable correspondence-search pipeline, and ODIN, a diffusion novel-view synthesis model trained on it with motion masking to generate navigable 3D scenes from a single image.
Problem
Real-world multi-view data for 3D learning is scarce because finding corresponding frames with diverse viewpoints in ordinary fixed-view video is sparse and computationally expensive, so single-image novel view synthesis for real scenes lags behind object-centric work.
Approach
Use 360° videos so frames can be rotated to overlap, search correspondences with DUSt3R confidence over short windows, propagate correspondences through a frame graph for long-range pairs, and train a latent diffusion NVS model with a learned motion mask that filters dynamic content from the loss.
Key results
- On Mip-NeRF 360, ODIN reaches LPIPS 0.587 versus 0.625 for ZeroNVS and 0.667 for Zero-1-to-3, with PSNR 16.84.
- On Google Scanned Objects 3D reconstruction, ODIN achieves Chamfer Distance 0.0697 and IoU 0.5328, the best among compared methods.
- On DTU, ODIN achieves LPIPS 0.378 and PSNR 16.67, best LPIPS and PSNR in the table, while SSIM is 0.525 (below SinNeRF's 0.560).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ODIN instead of ZeroNVS | Novel view synthesis · DTU | LPIPS: 0.380 -> 0.378 | Helped |
| Use ODIN instead of SinNeRF | Novel view synthesis · DTU | PSNR: 16.52 -> 16.67 | Helped |
| Use ODIN instead of SinNeRF | Novel view synthesis · DTU | SSIM: 0.560 -> 0.525 | Hurt |
| Use ODIN instead of ZeroNVS | Novel view synthesis · MipNeRF 360 | LPIPS: 0.625 -> 0.587 | Helped |
| Use ODIN instead of PixelNeRF | Novel view synthesis · MipNeRF 360 | PSNR: 16.50 -> 16.84 | Helped |
| Use ODIN instead of PixelNeRF | Novel view synthesis · MipNeRF 360 | SSIM: 0.556 -> 0.537 | Hurt |
| Use ODIN instead of Zero-1-to-3 | 3D reconstruction · Google Scanned Objects | Chamfer Distance: 0.0717 -> 0.0697 | Helped |
| Use ODIN instead of Zero-1-to-3 | 3D reconstruction · Google Scanned Objects | IoU: 0.5052 -> 0.5328 | Helped |
| Use ODIN instead of Zero 1-to-3 | 3D reconstruction · 360-1M held-out set | Chamfer Distance: 0.1059 -> 0.07992 | Helped |
| Use ODIN instead of Zero 1-to-3 | 3D reconstruction · 360-1M held-out set | IoU: 0.3178 -> 0.5267 | Helped |
| Increase sampling rate from 0.5 FPS to 1 FPS (ablation) | ODIN · NVS frame-rate ablation | LPIPS: 0.488 -> 0.467 | Helped |
| Increase sampling rate from 1 FPS to 5 FPS (ablation) | ODIN · NVS frame-rate ablation | LPIPS: 0.467 -> 0.461 | Helped |
Surprising findings
- Ablation shows that the motion-mask coefficient lambda strongly affects results, with lambda=1 best and lambda=0.1 giving PSNR 12.31.
Limitations
- Dynamic scene elements are filtered via motion masking rather than modeled, so the model does not capture object motion.
- Generated views can be inconsistent across the trajectory; 3D reconstruction relies on DUSt3R over generated images.
- Dataset is limited to YouTube 360° videos and may contain inappropriate or fake-enabling content; only video links are released.
Experimental setup
Datasets: 360-1M DTU Mip-NeRF 360 Google Scanned Objects Objaverse Co3D ACID RealEstate10K
Benchmarks: DTU Mip-NeRF 360 Google Scanned Objects 360-1M held-out set
Metrics: LPIPS PSNR SSIM Chamfer Distance IoU
Baselines: PixelNeRF SinNeRF DietNeRF NeRDi ZeroNVS Zero-1-to-3 MCC SJC-I Point-E
Builds on: Zero-1-to-3 DUSt3R Stable Diffusion COLMAP
Closest papers by meaning
- Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.89)
- OSN: Infinite Representations of Dynamic 3D Scenes from Monocular Videos (ICML 2024 · similarity 0.88)
- Leader360V: A Large-scale, Real-world 360 Video Dataset for Multi-task Learning in Diverse Environment (NeurIPS 2025 · similarity 0.88)
- MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views (NeurIPS 2024 · similarity 0.88)
- DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision (CVPR 2024 · similarity 0.88)
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.