Research map / 3D vision / 3D generation
Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Sparse3D distills a multiview-consistent Stable Diffusion controller (guided by epipolar features from sparse input views) into a NeRF using a new category-score distillation sampling (C-SDS) loss, improving novel-view synthesis and geometry from 2-6 posed views on CO3Dv2.
Problem
Reconstructing 3D objects and novel views from extremely sparse (2-3 or few) posed images, where generalizable NeRFs regress blurry outputs and SDS-based distillation yields blurry, oversaturated results with poor multiview consistency.
Approach
An epipolar feature renderer and a ControlNet-style epipolar controller guide a frozen Stable Diffusion v1.5 to produce multiview-consistent novel-view images; the NeRF is then optimized with C-SDS, which replaces the Gaussian noise term in SDS with the category-conditioned SD noise estimate, plus perceptual losses on one-step estimates.
Key results
- On unseen instances with 2 views, Sparse3D achieves LPIPS 0.22 and FID 147.65 versus SparseFusion's 0.23 and 187.22 (PSNR slightly lower at 20.95 vs 21.28).
- Geometry on unseen instances improves Chamfer Distance to 0.21 from SparseFusion's 0.27 and F-score to 0.32 from 0.23.
- On unseen categories with 2 views, FID drops to 164.30 (Sparse3D diffusion) and geometry CD 0.27 vs SparseFusion's 0.37 (F-score 0.28 vs 0.18).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Sparse3D instead of SparseFusion | CO3Dv2 novel-view synthesis, unseen instances with 2 views · CO3Dv2 unseen instances, 2 views | FID: 187.22 -> 147.65 | Helped |
| use Sparse3D instead of SparseFusion | CO3Dv2 novel-view synthesis, unseen categories with 2 views · CO3Dv2 unseen categories, 2 views | FID: 290.45 -> 164.30 | Helped |
| use Sparse3D instead of SparseFusion | CO3Dv2 geometry reconstruction · CO3Dv2 unseen instances | Chamfer Distance: 0.27 -> 0.21 | Helped |
| replace SDS with C-SDS (ablation) | Sparse3D NeRF reconstruction · CO3Dv2 core subset | LPIPS: 0.23 -> 0.20 | Helped |
| replace SDS with C-SDS (ablation) | Sparse3D NeRF reconstruction · CO3Dv2 core subset | Chamfer Distance: 0.21 -> 0.19 | Helped |
| replace SDS with C-SDS (ablation) | Sparse3D NeRF reconstruction · CO3Dv2 51 categories, 2 views | LPIPS: 0.22 -> 0.21 | Helped |
| add the epipolar constraint to the controller (ablation) | Sparse3D diffusion model · CO3Dv2 unseen instances and categories | LPIPS: 0.24 -> 0.21 | Helped |
| add Stable Diffusion priors (ablation) | Sparse3D diffusion model · CO3Dv2 unseen categories | LPIPS: 0.29 -> 0.24 | Helped |
Surprising findings
- PSNR is slightly lower for Sparse3D than baselines, which the authors attribute to blurry outputs favored by pixel-wise MSE.
Limitations
- Extremely partial observations of the object in input views cause failures.
- Janus (multi-face) artifacts and difficulty with thin structures or self-occluded parts.
- Relies on accurate camera poses, which are hard to estimate from extremely sparse views.
Experimental setup
Datasets: CO3Dv2
Benchmarks: CO3Dv2 fewview-dev
Metrics: PSNR SSIM LPIPS FID CLIP similarity DISTS Chamfer Distance F-score
Baselines: PixelNeRF ViewFormer EFT SparseFusion SDS
Builds on: SparseFusion ControlNet DreamFusion ProlificDreamer GPNR
Closest papers by meaning
- How to Use Diffusion Priors under Sparse Views? (NeurIPS 2024 · similarity 0.91)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.90)
- DIFIX3D+: Improving 3D Reconstructions with Single-Step Diffusion Models (CVPR 2025 · similarity 0.90)
- Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day (ICLR 2024 · similarity 0.90)
- SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images (CVPR 2025 · similarity 0.90)
- Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.