Research map / 3D vision / Gaussian splatting
How to Use Diffusion Priors under Sparse Views?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IPSM, a rectified score distillation method that uses inline pose-based warping priors from sparse views (with an inpainting diffusion model) to fix SDS mode deviation, plugged into a 3DGS pipeline (IPSM-Gaussian) for sparse-view novel view synthesis.
Problem
Score Distillation Sampling underperforms and can even hurt sparse-view 3D reconstruction because sparse views have low information entropy compared to text, causing mode deviation and unstable optimization.
Approach
Warp seen training views to pseudo unseen viewpoints using rendered depth and pose to form a rectified distribution (via an inpainting diffusion model sharing the SD VAE space), and decompose the SDS objective into two sub-objectives that pass through this intermediate distribution, without fine-tuning or pre-training the diffusion model.
Key results
- On LLFF 3-view, IPSM-Gaussian reaches SSIM 0.702, LPIPS 0.207, PSNR 20.44, AVGE 0.101, best among compared methods.
- On DTU 3-view, IPSM-Gaussian reaches SSIM 0.856, LPIPS 0.121, PSNR 19.99, AVGE 0.077; AVGE improves 23.76% over FSGS.
- Adding SDS (CFG=7.5) to the 3DGS base gives small SSIM gain but lower LPIPS/PSNR, while IPSM improves LPIPS by 9.84% and AVGE by 9.60% over base on LLFF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IPSM-Gaussian instead of FSGS | 3DGS sparse-view reconstruction · LLFF, 3 views | AVGE: +6.48% | Helped |
| Use IPSM-Gaussian instead of FSGS | 3DGS sparse-view reconstruction · DTU, 3 views | AVGE: +23.76% | Helped |
| Add IPSM regularization (G1 and G2) (ablation) | 3DGS sparse-view reconstruction · LLFF, 3 views | AVGE: 0.125 -> 0.113 | Helped |
| Add IPSM G2 to IPSM G1 (ablation) | IPSM-Gaussian · LLFF, 3 views | AVGE: 0.121 -> 0.113 | Helped |
| Add depth regularization (ablation) | IPSM-Gaussian with IPSM G1 and G2 · LLFF, 3 views | AVGE: 0.113 -> 0.104 | Helped |
| Add geometry consistency regularization (ablation) | IPSM-Gaussian with IPSM and depth regularization · LLFF, 3 views | AVGE: 0.104 -> 0.101 | Helped |
| Add SDS with CFG=7.5 | Base 3DGS · LLFF, 3 views | AVGE: 0.125 -> 0.128 | Hurt |
| Add SDS with CFG=100 | Base 3DGS · LLFF, 3 views | AVGE: 0.125 -> 0.162 | Hurt |
| Use IPSM instead of SDS with CFG=7.5 | 3DGS sparse-view reconstruction · LLFF, 3 views | AVGE: 0.128 -> 0.113 | Helped |
| Use depth regularization without other regularization (ablation) | Base 3DGS · LLFF, 3 views | AVGE: 0.125 -> 0.105 | Helped |
| Use geometry consistency regularization without other regularization (ablation) | Base 3DGS · LLFF, 3 views | AVGE: 0.125 -> 0.117 | Helped |
| Replace the Stable Diffusion Inpainting VAE with the original Stable Diffusion VAE (ablation) | SDS (CFG=7.5) · LLFF, 3 views | AVGE: 0.128 -> 0.133 | Mixed |
Surprising findings
- Vanilla SDS degrades sparse-view reconstruction relative to the baseline and increases training variance, as the paper's 'SDS dilemma'.
- Using the original Stable Diffusion VAE instead of the inpainting VAE gives similar SDS results, with CFG=7.5 even less stable.
Limitations
- The rectified distribution must share the feature space (VAE) of the pre-trained diffusion prior, restricting usable inpainting models.
- Warping quality depends on rendered depth accuracy; inline priors degrade with depth error, and SfM fails on some DTU scenes requiring random initialization.
Experimental setup
Datasets: LLFF DTU
Benchmarks: LLFF DTU
Metrics: SSIM LPIPS PSNR AVGE
Baselines: SRF PixelNeRF MVSNeRF Mip-NeRF DietNeRF RegNeRF FreeNeRF SparseNeRF 3DGS FSGS DNGaussian SDS
Builds on: 3D Gaussian Splatting Score Distillation Sampling FSGS DNGaussian
Closest papers by meaning
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.91)
- Taming Video Diffusion Prior with Scene-Grounding Guidance for 3D Gaussian Splatting from Sparse Inputs (CVPR 2025 · similarity 0.89)
- Provably Robust Score-Based Diffusion Posterior Sampling for Plug-and-Play Image Reconstruction (NeurIPS 2024 · similarity 0.87)
- Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.87)
- DiffusionBlend: Learning 3D Image Prior through Position-aware Diffusion Score Blending for 3D Computed Tomography Reconstruction (NeurIPS 2024 · similarity 0.87)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.