Research map / 3D vision / 3D generation

Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator

NeurIPS 2026 · 3D vision › 3D generation · approach: Generative indoor scene synthesis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Image2Sim, a real-time neural simulator that builds interactive scenes from posed RGB-D sequences via feed-forward feature Gaussians and a one-step MeanFlow pixel renderer, and uses it as an automated data engine to train a VLN navigation model (Image2Nav).

Problem

Embodied navigation lacks scalable, high-fidelity, physically grounded interactive environments; real scans are small-scale and synthetic simulators have large sim-to-real gaps.

Approach

Decouple 3D spatial anchoring (feed-forward feature Gaussians lifted from posed RGB-D in one pass) from photorealistic observation synthesis (a geometry-aware one-step pixel-flow renderer trained with MeanFlow, alpha-gated skips, and momentum self-distillation), then automatically generate trajectories, observations and VLM-annotated instructions at scale.

Key results

• Image2Nav (180° FOV) reaches 70.3 SR / 65.6 SPL on R2R-CE Val Unseen zero-shot in Habitat, above EfficientVLN (64.2 SR) and DualVLN (64.3 SR).
• Image2Sim renders panoramic RGB-D at about 41-46 FPS on a single RTX 4090 with 17.43 PSNR on noisy RealSee3D-Real, versus 0.3 FPS for DiT360 and 3.4 FPS for SE3DS.
• Real-world Hello Robot Stretch 3 trials: Image2Nav reaches goal-oriented SR 9/20 versus 5/20 for the strongest baseline (DualVLN).

Surprising findings

• Image2Nav trained only on human-annotated R2R and RxR data performs well below most competing baselines; its advantage comes from the 10M synthetic samples.
• Feed-forward AnySplat renders much faster (>115 FPS) than Image2Sim but yields lower PSNR, showing a fidelity-speed trade-off.

Limitations

• Compact renderer trades off model capacity and generation quality for real-time speed and online training.
• Simulator handles only navigation-level physical validity; no contact dynamics, movable objects, or human-robot interaction.
• VLM-based instruction annotation may introduce linguistic bias or semantic mismatch.

Experimental setup

Datasets: RealSee3D Structured3D ARKitScenes HM3D ScanNet Gibson Matterport3D Matterport3D-360 R2R-CE RxR-CE

Benchmarks: R2R-CE Val Unseen RxR-CE Val Unseen REVERIE-CE Val Unseen RealSee3D-Synthesis Matterport3D-360 RealSee3D-Real

Metrics: LPIPS PSNR SSIM FPS NE OSR SR SPL nDTW real-world OSR/SR

Baselines: DiT360 SE3DS AnySplat NaVid Uni-NaVid NaVILA Dynam3D NavFoM StreamVLN D3D-VLP JanusVLN InternVLA-N1

Builds on: 3D Gaussian Splatting MeanFlow DINOv3 DINO DAgger Habitat

Closest papers by meaning

• Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation (CVPR 2025 · similarity 0.91)
• Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation (CVPR 2024 · similarity 0.90)
• Scene Map-based Prompt Tuning for Navigation Instruction Generation (CVPR 2025 · similarity 0.89)
• RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigation (CVPR 2025 · similarity 0.88)
• NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.88)
• InternScenes: A Large-scale Simulatable Indoor Scene Dataset with Realistic Layouts (NeurIPS 2025 · similarity 0.88)