Research map / 3D vision / 3D generation
NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free novel view synthesis that adaptively modulates the score function of a pretrained video diffusion model (Stable Video Diffusion) with depth-warped input views, using an error-bound-derived adaptive weight.
Problem
Novel view synthesis from single or multiple views of static scenes or monocular videos of dynamic scenes, with limited input views and without task-specific training.
Approach
Reformulates NVS as guided diffusion sampling: the clean-image estimate of a pretrained video diffusion model is modulated with depth-warped input views, with a per-step weight lambda(t, pose) chosen to minimize a theoretical estimation error bound (depending on noise level and pose change), and applied either by direct replacement (DGS) or by posterior gradient-based updates (Post).
Key results
- On single-view NVS, Ours (Post) reaches FID 165.12 and ATE 0.767 versus MotionCtrl's 179.24 and 3.851.
- On monocular dynamic-scene NVS, Ours (DGS) achieves FID 37.973 versus 4D-Gaussian's 74.34 and MotionCtrl's 70.35.
- Posterior sampling yields much more accurate poses than DGS (single-view ATE 0.767 vs 4.533) but takes about 1 hour for 25 views versus 6 minutes for DGS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase reverse diffusion steps from 25 to 100 (ablation) | Ours (Post) single view · Tanks and Temples single view | ATE: 5.317 to 0.767 | Helped |
| Increase reverse diffusion steps from 25 to 100 (ablation) | Ours (Post) single view · Tanks and Temples single view | FID: 175.432 to 165.12 | Helped |
| Adaptive lambda vs. constant lambda 0.5 (ablation) | Ours single view · Tanks and Temples single view | FID: 175.68 to 165.12 | Helped |
| Adaptive lambda vs. linear and exponential lambda schedules (ablation) | Ours single view · Tanks and Temples single view | ATE: 1.04 (linear) and 1.363 (exponential) to 0.767 | Helped |
| Gradient normalization in posterior sampling update (ablation) | Ours (Post) · Tanks and Temples single view | ATE: 4.71 without normalization to 0.767 with | Helped |
| Gradient updating rate kappa=0.5 instead of 2e-2*sqrt(sigma) (ablation) | Ours (Post) · Tanks and Temples single view | FID: 165.12 to 268.17 | Hurt |
| Posterior sampling instead of directly guided sampling (ablation) | SVD-based NVS · Tanks and Temples single and multi-view | ATE: single view ATE 0.767 vs 4.533 (DGS); multi-view 4.052 vs 22.00 | Helped |
| Posterior sampling instead of directly guided sampling (ablation) | SVD-based NVS · Tanks and Temples single view | FID: 165.12 (Post) vs 166.50 (DGS) | No clear effect |
| Adaptive modulation (Ours) vs. replacing prior with warped view (lambda to infinity) on single view (ablation) | SVD single-view NVS · Tanks and Temples single view | qualitative warping error correction | Helped |
| Ours (Post) vs MotionCtrl on single view | SVD-based NVS · Tanks and Temples single view | FID: 165.12 vs 179.24 | Helped |
| Ours (Post) vs MotionCtrl on monocular dynamic videos | SVD-based NVS · Dynamic monocular video | FID: 39.86 vs 70.35 FID; ATE 2.308 vs 3.384 | Mixed |
| Depth estimator swap: DepthAnything V1 vs V2 vs DINOv2 in full pipeline (ablation) | Ours (Post) single view · Tanks and Temples single view | FID: 165.12 (V1), 162.896 (V2), 158.24 (DINOv2) | No clear effect |
Surprising findings
- Directly replacing the score (DGS) gives comparable FID to posterior sampling but markedly worse pose accuracy, and a constant or linear/exponential lambda schedule degrades results relative to the derived adaptive weight.
Limitations
- Substantially slower than existing NVS methods, especially posterior sampling.
- Relies on estimated depth; absolute scale is unreliable for single views and monocular videos, so only paired metrics like LPIPS are compared in the appendix.
- Some baseline results are incomplete (marked with *) and some metrics are not computable for certain methods, complicating comparison.
Experimental setup
Datasets: Tanks and Temples DTU DyCheck YouTube videos (9 clips, 25 frames each) Internet images
Benchmarks: Tanks and Temples DTU YouTube dynamic videos DyCheck
Metrics: FID ATE RPE-T RPE-R LPIPS PSNR SSIM
Baselines: Sparse Gaussian Sparse NeRF Text2Nerf Photoconsistent-NVS 3D-aware MotionCtrl Deformable-Gaussian 4D-Gaussian
Builds on: Stable Video Diffusion EDM DDIM Depth Anything
Closest papers by meaning
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.93)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.91)
- Dynamic View Synthesis as an Inverse Problem (NeurIPS 2025 · similarity 0.90)
- Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day (ICLR 2024 · similarity 0.89)
- Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos (NeurIPS 2025 · similarity 0.89)
- ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.