Research map / 3D vision / 3D generation
Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Vid2Sim converts monocular video into a photorealistic, physically interactive Gaussian-splat-plus-mesh simulator in which navigation agents are trained with SAC and transferred zero-shot to a real delivery robot.
Problem
The sim-to-real gap for urban navigation agents, since traditional simulators and mesh-based graphics give unrealistic observations and limited environment diversity.
Approach
Reconstruct urban scenes from casual monocular video with scale-invariant monocular depth/normal supervision and a geometry-consistent loss on Gaussian Splatting, pair the GS rendering (for visuals) with a TSDF-extracted invisible mesh (for collisions) in Unity, and augment with obstacles, pedestrians, relighting, and weather.
Key results
- Vid2Sim achieves 32.4 PSNR / 0.927 SSIM / 0.127 LPIPS on reconstruction, beating 3DGS (31.85) and 2DGS (30.82).
- Agents trained with static and dynamic obstacles reach 81.6% SR on PointNav and 74.4% on SocialNav in simulation, versus 48.8% and 43.2% for mesh-based RGB.
- Zero-shot real-world success rate is 85%, 65%, and 55% for Go Straight, Static Obstacle, and Dynamic Obstacle when trained on 30 environments, versus 0% for the mesh baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Vid2Sim instead of 3DGS for scene reconstruction | scene reconstruction · Vid2Sim dataset test scenes | PSNR: 31.85 -> 32.41 | Helped |
| use Vid2Sim instead of mesh-based visual observations | urban navigation agent · PointNav | success rate: 48.8% -> 81.6% | Helped |
| use Vid2Sim instead of mesh-based visual observations | urban navigation agent · SocialNav | success rate: 43.2% -> 74.4% | Helped |
| add static and dynamic obstacles during training (ablation) | Vid2Sim navigation agent · PointNav | success rate: 68.8% -> 81.6% | Helped |
| add static and dynamic obstacles during training (ablation) | Vid2Sim navigation agent · SocialNav | success rate: 61.6% -> 74.4% | Helped |
| add static obstacles during training (ablation) | Vid2Sim navigation agent · PointNav | success rate: 68.8% -> 80.8% | Helped |
| add static obstacles during training (ablation) | Vid2Sim navigation agent · SocialNav | success rate: 61.6% -> 71.2% | Helped |
| add dynamic pedestrians to training with static obstacles (ablation) | Vid2Sim navigation agent · PointNav | success rate: 80.8% -> 81.6% | Helped |
| add dynamic pedestrians to training with static obstacles (ablation) | Vid2Sim navigation agent · SocialNav | success rate: 71.2% -> 74.4% | Helped |
| train on 30 Vid2Sim environments instead of the mesh-based baseline | zero-shot real-world navigation agent · Go Straight | success rate: 0% -> 85% | Helped |
| train on 30 Vid2Sim environments instead of the mesh-based baseline | zero-shot real-world navigation agent · Static Obstacle | success rate: 0% -> 65% | Helped |
| train on 30 Vid2Sim environments instead of the mesh-based baseline | zero-shot real-world navigation agent · Dynamic Obstacle | success rate: 0% -> 55% | Helped |
Surprising findings
- The mesh-based baseline fails all three real-world tasks (0% success), indicating a large sim-to-real gap for mesh RGB observations.
Limitations
- Mesh baseline is a simulated approximation of Video2Game rather than the original system
- Real-world evaluation uses a limited number of trials (20 per task) on one robot
Experimental setup
Datasets: Vid2Sim dataset (30 scenes from web-sourced 15-second videos)
Benchmarks: Vid2Sim PointNav Vid2Sim SocialNav real-world delivery robot tasks
Metrics: PSNR SSIM LPIPS success rate SPL cost (hit number) social navigation score
Baselines: Instant-NGP 3DGS 2DGS Video2Game Mesh-based simulation (RGB)
Builds on: 3D Gaussian Splatting 2DGS Video2Game Soft Actor-Critic TSDF fusion
Closest papers by meaning
- Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator (NeurIPS 2026 · similarity 0.91)
- Learning to Navigate Efficiently and Precisely in Real Environments (CVPR 2024 · similarity 0.87)
- Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration (NeurIPS 2025 · similarity 0.87)
- Learning Interactive Real-World Simulators (ICLR 2024 · similarity 0.85)
- Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation (CVPR 2024 · similarity 0.85)
- Navigation World Models (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.