Research map / 3D vision / Gaussian splatting
Taming Video Diffusion Prior with Scene-Grounding Guidance for 3D Gaussian Splatting from Sparse Inputs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free scene-grounding guidance that steers a camera-controlled video diffusion model with renderings from an optimized 3DGS, plus trajectory initialization and a tailored 3DGS optimization scheme, to address extrapolation and occlusion in sparse-input 3DGS.
Problem
Sparse-input 3D Gaussian Splatting suffers from unobserved regions outside the field of view (extrapolation) and occluded regions for novel views, causing holes; generated video sequences from diffusion models are inconsistent and degrade reconstruction.
Approach
Use renderings from an optimized baseline 3DGS as a training-free guidance target (masked L1 plus perceptual loss on the predicted clean latent) during video diffusion denoising, so generated sequences are consistent and scene-grounded; trajectories are chosen where baseline renderings have holes.
Key results
- On Replica, the method reaches 26.35 PSNR / 0.872 SSIM / 0.122 LPIPS versus 22.99 PSNR for FSGS with DUSt3R initialization (FSGS↕).
- On ScanNet++, the method reaches 23.89 PSNR versus 21.23 for FSGS↕, a gain of over 2.5 dB.
- Vanilla generation alone raises full-image PSNR from 22.80 to 23.69 but lowers observable-region PSNR from 25.45 to 25.00; scene-grounding guidance raises observable-region PSNR to 26.52.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use video-diffusion-generated sequences without scene-grounding guidance (ablation) | Baseline 3DGS · Replica | PSNR: 22.80 -> 23.69; observable regions 25.45 -> 25.00 | Mixed |
| add scene-grounding guidance to generated sequences (ablation) | Baseline 3DGS with vanilla generation · Replica | PSNR: 23.69 -> 25.03 | Helped |
| add trajectory initialization (ablation) | guided generation for Baseline 3DGS · Replica | PSNR: 25.03 -> 25.58 | Helped |
| add perceptual loss for generated-view supervision (ablation) | Baseline 3DGS with guided generation and trajectory initialization · Replica | PSNR: 25.58 -> 26.35 | Helped |
| remove local sampling of generated views (ablation) | Baseline 3DGS with the full optimization scheme · Replica | PSNR: 26.35 -> 26.28 | Hurt |
| remove the global list of generated views (ablation) | Baseline 3DGS with the full optimization scheme · Replica | PSNR: 26.35 -> 26.01 | Hurt |
| use the proposed reconstruction-by-generation method instead of FSGS↕ | 3DGS initialized with DUSt3R point cloud · Replica | PSNR: 22.99 -> 26.35 | Helped |
| use the proposed reconstruction-by-generation method instead of 3DGS↕ | 3DGS initialized with DUSt3R point cloud · ScanNet++ | PSNR: 21.41 -> 23.89 | Helped |
| use trajectory initialization with SDS and SDInpaint (ablation) | Baseline 3DGS+SDS with SDInpaint · Replica | PSNR: 24.33 -> 25.15 | Helped |
Surprising findings
- Vanilla generated sequences degrade observable-region quality even though full-image metrics improve.
Limitations
- Guidance target is the rendered sequence from a baseline 3DGS, which is imperfect and can propagate its errors.
- Evaluation limited to six Replica and four ScanNet++ scenes with indoor settings.
- Requires per-iteration denoising guidance with gradients through the VAE decoder and diffusion model, which adds cost.
Experimental setup
Datasets: Replica ScanNet++
Benchmarks: Replica ScanNet++
Metrics: PSNR SSIM LPIPS
Baselines: Mip-NeRF InfoNeRF DietNeRF FreeNeRF 3DGS DNGaussian FSGS LaMa inpainting SDS with SDInpaint
Builds on: 3D Gaussian Splatting FSGS DUSt3R training-free guidance for diffusion models camera-controlled image-to-video diffusion
Closest papers by meaning
- 3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors (NeurIPS 2024 · similarity 0.92)
- Generative Sparse-View Gaussian Splatting (CVPR 2025 · similarity 0.91)
- Wonderland: Navigating 3D Scenes from a Single Image (CVPR 2025 · similarity 0.91)
- Novel View Synthesis from A Few Glimpses via Test-Time Natural Video Completion (NeurIPS 2025 · similarity 0.90)
- Quantifying and Alleviating Co-Adaptation in Sparse-View 3D Gaussian Splatting (NeurIPS 2025 · similarity 0.90)
- DropGaussian: Structural Regularization for Sparse-view Gaussian Splatting (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.