Research map / 3D vision / NeRF and radiance fields
Video2Game: Real-time Interactive Realistic and Browser-Compatible Environment from a Single Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Converts a single real-world video into a browser-compatible, real-time interactive game environment via large-scale NeRF, NeRF-to-neural-textured-mesh baking, and rigid-body physics.
Problem
Building realistic, interactive virtual environments from real-world footage is costly; NeRFs render photorealistic views but are too slow for real-time games and lack physical interaction and game-engine compatibility.
Approach
Combines a large-scale depth/normal/semantic-regularized NeRF with a NeRF-baked neural-textured mesh (UV neural texture plus GLSL MLP shader), scene decomposition into actionable entities, and rigid-body physics in a WebGL engine.
Key results
- On KITTI-360 the mesh model reaches 23.35 PSNR versus 19.67 for MobileNeRF and 22.37 for BakedSDF, while the base NeRF reaches 27.62 PSNR versus 27.46 for Instant-NGP.
- Geometry outlier rate on KITTI-360 is 13.23% for the proposed NeRF versus 22.89% for Instant-NGP and 43.78% for BakedSDF.
- The browser-based game runs at 102 to 240 FPS across tested Mac, Intel and AMD platforms.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add regularization terms (ablation) | Video2Game NeRF · KITTI-360 | PSNR: 27.46 -> 27.52 | Helped |
| Add regularization terms (ablation) | Video2Game NeRF · KITTI-360 | SSIM: 0.853 -> 0.861 | Helped |
| Add regularization terms (ablation) | Video2Game NeRF · KITTI-360 | LPIPS: 0.165 -> 0.157 | Helped |
| Add monocular cues (ablation) | Video2Game NeRF · KITTI-360 | PSNR: 27.52 -> 27.62 | Helped |
| Add monocular cues (ablation) | Video2Game NeRF · KITTI-360 | SSIM: 0.861 -> 0.871 | Helped |
| Add monocular cues (ablation) | Video2Game NeRF · KITTI-360 | LPIPS: 0.157 -> 0.131 | Helped |
| Add regularization terms (ablation) | Video2Game mesh rasterization · KITTI-360 | PSNR: 22.54 -> 22.97 | Helped |
| Add regularization terms (ablation) | Video2Game mesh rasterization · KITTI-360 | SSIM: 0.716 -> 0.732 | Helped |
| Add regularization terms (ablation) | Video2Game mesh rasterization · KITTI-360 | LPIPS: 0.350 -> 0.303 | Helped |
| Add monocular cues (ablation) | Video2Game mesh rasterization · KITTI-360 | PSNR: 22.97 -> 23.35 | Helped |
| Add monocular cues (ablation) | Video2Game mesh rasterization · KITTI-360 | SSIM: 0.732 -> 0.765 | Helped |
| Add monocular cues (ablation) | Video2Game mesh rasterization · KITTI-360 | LPIPS: 0.303 -> 0.246 | Helped |
Surprising findings
- Baking NeRF into a mesh lowers rendering quality relative to volume rendering (KITTI-360 PSNR drops from 27.62 to 23.35).
Limitations
- Mesh baking reduces rendering quality compared with the underlying volume-rendered NeRF.
- Physics limited to rigid-body dynamics; physical parameters are set manually or estimated by LLMs.
- Grasping is simplified by automatically picking up objects near the gripper rather than simulating real grasps.
Experimental setup
Datasets: Gardenvase KITTI-360 VR-NeRF
Benchmarks: Gardenvase KITTI-360 VR-NeRF
Metrics: PSNR SSIM LPIPS RMSE MAE outlier rate FPS
Baselines: Instant-NGP Nerfacto 3D Gaussian Splatting MobileNeRF BakedSDF
Builds on: Instant-NGP MonoSDF BlockNeRF NeRF2Mesh MobileNeRF
Closest papers by meaning
- DRAWER: Digital Reconstruction and Articulation With Environment Realism (CVPR 2025 · similarity 0.88)
- VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams (CVPR 2024 · similarity 0.87)
- The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control (NeurIPS 2025 · similarity 0.86)
- HoloScene: Simulation‑Ready Interactive 3D Worlds from a Single Video (NeurIPS 2025 · similarity 0.85)
- LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Control and Rendering (NeurIPS 2024 · similarity 0.85)
- Diffusion Models Are Real-Time Game Engines (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.