Research map / 3D vision / 3D generation
SimVS: Simulating World Inconsistencies for Robust View Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses a video diffusion model (Lumiere) to simulate dynamic and lighting inconsistencies on multiview datasets, then fine-tunes CAT3D into a multiview harmonization model that maps sparse inconsistent captures to consistent views for robust NeRF-based view synthesis.
Problem
Novel view synthesis from sparse casual captures that contain scene motion and lighting changes, where existing sparse-view methods assume consistent inputs and dense robust methods need hundreds of images.
Approach
Generate paired inconsistent/consistent training data by prompting an image-to-video diffusion model to simulate scene motion and lighting changes from static-camera frames, then train a multiview diffusion harmonization model that reconciles inconsistent conditioning images into a single consistent state given a reference image.
Key results
- On DyCheck, the model reaches 16.73 PSNR / 0.413 LPIPS versus 15.59 PSNR / 0.462 LPIPS for CAT3D (all images).
- On the lighting dataset, PSNR is 20.98 versus 18.26 for CAT3D (all images) and 14.80 for WildGaussians.
- Heuristic flow-field augmentation (15.52 PSNR) and animated Objaverse data (14.92 PSNR) underperform the video-model-simulated data (16.60 PSNR) on dynamics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SimVS harmonization instead of CAT3D (all images) | Sparse view synthesis · DyCheck | PSNR: 15.59 -> 16.73 | Helped |
| use SimVS harmonization instead of CAT3D (all images) | Sparse view synthesis · Lighting dataset | PSNR: 18.26 -> 20.98 | Helped |
| replace video-model dynamics simulation with sparse-flow heuristic augmentation (ablation) | SimVS · DyCheck | PSNR: 16.60 -> 15.52 | Hurt |
| replace video-model dynamics simulation with animated Objaverse assets (ablation) | SimVS · DyCheck | PSNR: 16.60 -> 14.92 | Hurt |
| replace video-model lighting simulation with inconsistent color tints (ablation) | SimVS · Lighting dataset | PSNR: 20.98 -> 18.96 | Hurt |
Surprising findings
- Heuristic tint augmentation slightly outperforms vanilla CAT3D on lighting, suggesting the model mainly learns to match mean color.
Limitations
- Requires accurate camera poses, which are hard to obtain for sparse captures with COLMAP.
- Degenerate or hallucinated content when inconsistency is large or views have little overlap.
- Lumiere struggles with lighting changes under non-generic prompts, requiring predetermined prompts instead of Gemini-generated ones for lighting.
Experimental setup
Datasets: Mannequin Challenge RealEstate10k Objaverse DyCheck
Benchmarks: DyCheck Custom lighting dataset
Metrics: PSNR SSIM LPIPS
Baselines: CAT3D (single image) CAT3D (all images) WildGaussians Heuristic Augmentation Animated Objaverse
Builds on: CAT3D Lumiere Zip-NeRF COLMAP
Closest papers by meaning
- SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints (ICLR 2025 · similarity 0.89)
- Video Perception Models for 3D Scene Synthesis (NeurIPS 2025 · similarity 0.88)
- World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.87)
- PanoWorld: Geometry-Consistent Panoramic Video World Modeling (NeurIPS 2026 · similarity 0.87)
- Geometry-guided Online 3D Video Synthesis with Multi-View Temporal Consistency (CVPR 2025 · similarity 0.87)
- GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.