Research map / 3D vision / 3D generation
The More You See in 2D the More You Perceive in 3D
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SAP3D, which test-time fine-tunes a pretrained view-conditioned diffusion model (Zero-1-to-3) and refines camera poses from a few unposed images, then distills into a NeRF for 3D reconstruction and novel view synthesis that improves as more views are added.
Problem
3D reconstruction and novel view synthesis from a variable number of unposed images, where prior single-view diffusion-based methods cannot use extra views and multi-view methods need accurate poses and many views.
Approach
Adapt a pretrained view-conditioned diffusion model at test time on all pairs of input images while jointly optimizing their relative camera poses, with a 3D prior preservation loss; the instance-specific model and poses then serve as priors for NeRF reconstruction via SDS plus photometric losses.
Key results
- On GSO with 20 objects, 3D reconstruction Chamfer Distance drops from 0.168 (1 view, Zero-1-to-3) to 0.019 (5 views) and VolumeIoU rises from 0.25 to 0.68.
- 2D novel view synthesis PSNR improves from 15.2 (1 view) to 19.5 (5 views) on GSO, without saturating at 5 images.
- Test-time adaptation improves 3D reconstruction over the no-adaptation variant (CD 0.029 to 0.015, F1@0.04 0.87 to 0.94) and reduces camera rotation error with more views.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase input views from 1 to 2 | SAP3D · GSO 3D reconstruction | PSNR: 14.1 -> 18.0 | Helped |
| increase input views from 1 to 5 | SAP3D · GSO 3D reconstruction | CD: 0.168 -> 0.019 | Helped |
| increase input views from 1 to 5 | SAP3D · GSO novel-view synthesis | PSNR: 15.2 -> 19.5 | Helped |
| replace RelPose++ camera initialization with RelPose++* trained on Objaverse (ablation) | SAP3D · GSO 3D reconstruction | PSNR: 13.9 -> 17.1 | Helped |
| replace RelPose++ camera initialization with RelPose++* trained on Objaverse (ablation) | SAP3D · GSO novel-view synthesis | PSNR: 13.3 -> 16.3 | Helped |
| add test-time adaptation of the diffusion model and camera poses (ablation) | SAP3D · GSO 3D reconstruction | CD: 0.029 -> 0.015 | Helped |
| add test-time adaptation of the diffusion model and camera poses (ablation) | SAP3D · GSO novel-view synthesis | PSNR: 16.3 -> 18.2 | Helped |
| add the SDS loss to 3D reconstruction (ablation) | SAP3D · GSO 3D reconstruction | PSNR: 9.8 -> 18.1 | Helped |
| add the photometric data loss to 3D reconstruction (ablation) | SAP3D · GSO 3D reconstruction | PSNR: 15.3 -> 18.1 | Helped |
Surprising findings
- Camera-pose refinement via test-time adaptation gives the largest gain between one and two views, after which improvements diminish and saturate around four views for 3D reconstruction.
Limitations
- Optimization-based pipeline requires about 15 minutes per object of test-time fine-tuning, so not real-time.
- Camera pose parameterization inherited from Zero-1-to-3 has only 3 degrees of freedom.
- Relies on several independent pretrained components rather than an end-to-end model.
Experimental setup
Datasets: Objaverse Co3D GSO LAION
Benchmarks: Google Scanned Objects (GSO) real captured objects (qualitative)
Metrics: PSNR SSIM LPIPS Chamfer Distance F1 score VolumeIoU relative rotation error
Baselines: Zero-1-to-3 RelPose++ SAP3D w/o adaptation PixelNeRF SyncDreamer
Builds on: Zero-1-to-3 RelPose++ DreamBooth score distillation sampling stochastic conditioning
Closest papers by meaning
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.89)
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.89)
- Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day (ICLR 2024 · similarity 0.88)
- Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion (CVPR 2025 · similarity 0.88)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.88)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.