Research map / Video / Video generation
Martian World Model: Controllable Video Synthesis with Physically Accurate 3D Reconstructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds M3arsSynth, a pipeline that reconstructs metric-scale 3D Martian scenes from NASA rover stereo imagery to render multimodal video/depth/normal/pose/text data, and fine-tunes CogVideoX-5B-I2V into MarsGen, a camera-controllable Martian video generator.
Problem
Synthesizing realistic, 3D-consistent Martian landscape videos for mission rehearsal and robotic simulation is hard due to scarce Martian data, sparse and photometrically inconsistent rover stereo imagery, and a domain gap from terrestrial video models.
Approach
Use geometric foundation models (VGGT for intrinsics, metric depth, GIM matching, PnP, depth rescaling) plus 3D Gaussian Splatting with bilateral grids to build a metric 3D Mars dataset, then fine-tune a camera-controlled image-to-video diffusion transformer on rendered Mars videos.
Key results
- MarsGen achieves FID 38.779 and FVD 364.822 versus CogVideoX's 48.912 and 411.808 in the image-to-video comparison.
- MarsGen reaches Warp Error 6.071 (vs CogVideoX 6.866) and novel-view PSNR 21.239 (vs CameraCtrl 20.014).
- The metric-aware reconstruction pipeline achieves 100% data utilization and 0.77 px reprojection error, versus COLMAP at 71.8% utilization and MASt3R at 46.98 px.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CogVideoX-5B-I2V on M3arsSynth for Martian video generation | MarsGen · Martian video generation | FID: 48.912 -> 38.779 | Helped |
| Fine-tune CogVideoX-5B-I2V on M3arsSynth for Martian video generation | MarsGen · Martian video generation | FVD: 411.808 -> 364.822 | Helped |
| Fine-tune CogVideoX-5B-I2V on M3arsSynth for Martian video generation | MarsGen · Martian video generation | Warp Error: 6.866 -> 6.071 | Helped |
| Fine-tune CogVideoX-5B-I2V on M3arsSynth for Martian video generation | MarsGen · Martian video generation | PSNR: 20.014 -> 21.239 | Helped |
| Fine-tune CogVideoX-5B-I2V on M3arsSynth for Martian video generation | MarsGen · Martian video generation | SSIM: 0.441 -> 0.614 | Helped |
| Fine-tune CogVideoX-5B-I2V on M3arsSynth for Martian video generation | MarsGen · Martian video generation | LPIPS: 0.408 -> 0.351 | Helped |
| Add depth rescaling (ablation) | M3arsSynth reconstruction pipeline · Novel view synthesis | PSNR: 32.20 -> 32.73 | Helped |
| Add depth rescaling (ablation) | M3arsSynth reconstruction pipeline · Novel view synthesis | LPIPS: 0.10 -> 0.09 | Helped |
| Use metric-aware initialization instead of MASt3R + 3DGS (ablation) | M3arsSynth reconstruction pipeline · Novel view synthesis | SSIM: 0.59 -> 0.93 | Helped |
| Use metric-aware initialization instead of MASt3R + 3DGS (ablation) | M3arsSynth reconstruction pipeline · Novel view synthesis | LPIPS: 0.24 -> 0.09 | Helped |
| Use metric-aware initialization rather than MASt3R(full) (ablation) | M3arsSynth reconstruction pipeline · Martian stereo imagery | 2D reprojection error (px): 46.980 -> 0.770 | Helped |
| Use VGGT intrinsics rather than MASt3R intrinsics (ablation) | M3arsSynth reconstruction pipeline · Martian stereo imagery | 2D reprojection error (px): 2.016 -> 0.770 | Helped |
Surprising findings
- VGGT point clouds from the Martian inputs show significant artifacts, so the pipeline uses VGGT only for intrinsics.
- COLMAP fails on nearly 30% of preprocessed Martian image pairs.
Limitations
- Current work does not model predictive temporal environmental dynamics
- Lacks fine-grained 3D semantic understanding
- Mars-specific evaluation is on the authors' own generated data with limited external validation
Experimental setup
Datasets: NASA PDS Martian rover stereo navigation imagery M3arsSynth
Benchmarks: M3arsSynth held-out videos
Metrics: FID FVD 2D Warp Error PSNR SSIM LPIPS Reprojection error
Baselines: Pyramidal-Flow CogVideoX Kling Sora CameraCtrl ViewCrafter COLMAP MASt3R
Builds on: CogVideoX 3D Gaussian Splatting ControlNet VGGT GIM
Closest papers by meaning
- SimVS: Simulating World Inconsistencies for Robust View Synthesis (CVPR 2025 · similarity 0.85)
- Mars-Bench: A Benchmark for Evaluating Foundation Models for Mars Science Tasks (NeurIPS 2025 · similarity 0.84)
- PanoWorld: Geometry-Consistent Panoramic Video World Modeling (NeurIPS 2026 · similarity 0.83)
- AerialMegaDepth: Learning Aerial-Ground Reconstruction and View Synthesis (CVPR 2025 · similarity 0.83)
- Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation (CVPR 2025 · similarity 0.82)
- RoboScape: Physics-informed Embodied World Model (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.