Research map / 3D vision / 3D generation
DiffusionSfM: Predicting Structure and Motion via Ray Origin and Endpoint Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffusionSfM, a transformer-based denoising diffusion model that jointly predicts per-pixel ray origins and endpoints in a global frame to recover cameras and geometry from unordered multi-view images without global optimization.
Problem
Structure-from-Motion typically uses a two-stage pipeline (pairwise reasoning plus global optimization); the paper targets end-to-end multi-view prediction of both 3D structure and camera poses, with uncertainty modeling, under sparse views.
Approach
Represent scene geometry and cameras as pixel-wise ray origins and endpoints in a global frame, predicted by a DiT-based diffusion model with DINOv2 features and a DPT decoder; handle missing depth via GT mask conditioning and unbounded coordinates via homogeneous coordinates; use sparse-to-dense training.
Key results
- On CO3D seen categories with 8 views, DiffusionSfM achieves 95.2% rotation accuracy at 15° and 90.1% center accuracy at 0.1, versus DUSt3R's 94.3% and 76.8%.
- On CO3D unseen categories, DiffusionSfM reaches 84.1% center accuracy at 5 views versus DUSt3R's 71.7%, while DUSt3R keeps slightly higher rotation accuracy in some settings.
- Inference with 10 denoising steps takes 1.91 seconds on a single A5000 GPU with 8 views, versus 8.73 seconds for DUSt3R with global alignment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| GT mask conditioning (vs. nearest-neighbor depth interpolation) (ablation) | DiffusionSfM* on CO3D · CO3D unseen categories, Chamfer Distance | Chamfer Distance (lower better): 0.021 vs 0.024 at 8 views | Helped |
| Homogeneous coordinates for ray origins/endpoints (vs. standard R^3 coordinates) (ablation) | DiffusionSfM · Training stability | convergence | Helped |
| Sparse-to-dense training (initialize dense model from sparse model) (ablation) | DiffusionSfM · CO3D | performance | Helped |
| Early-stopping inference using x0-prediction at 10 denoising steps | DiffusionSfM · Inference speed, 8 views | runtime (s): 1.91 s vs 8.73 s for DUSt3R | Helped |
| Diffusion-based model vs. DUSt3R regression on CO3D camera center accuracy | DiffusionSfM · CO3D seen and unseen categories | Center Accuracy (@0.1): 90.1 vs 76.8 (seen, 8 views) | Helped |
| Diffusion model vs. DUSt3R on CO3D rotation accuracy | DiffusionSfM · CO3D unseen categories | Rotation Accuracy (@15°): 94.4 vs 93.4 at 8 views; DUSt3R slightly higher at some view counts | Mixed |
| Chamfer distance with masking: DUSt3R vs DiffusionSfM | DiffusionSfM · CO3D unseen categories, foreground points | Chamfer Distance: DUSt3R best with masking; DiffusionSfM 0.021 vs DUSt3R 0.020 at 8 views (all points) | Hurt |
| Naive RayDiffusion + MoGe depth baseline for geometry | RayDiffusion pose + MoGe depth · CO3D unseen categories, Chamfer Distance | Chamfer Distance: RD*+MoGe ~0.061 at 8 views vs DiffusionSfM* 0.021 | Hurt |
Surprising findings
- Removing GT mask conditioning and filling missing depth by interpolation degrades geometry quality across all view counts, even with masked loss.
- Early-stopping x0-predictions from early denoising timesteps give more accurate outputs than final-step diffusion outputs.
Limitations
- Pixel-space diffusion may need more capacity, causing noisy patterns at object boundaries.
- Attention cost scales quadratically with number of input images, requiring masked attention for large sets.
- Evaluated mainly with 2-8 input views; DUSt3R retains higher rotation accuracy on some unseen-category settings.
Experimental setup
Datasets: CO3D Habitat ScanNet++ ArkitScenes Static Scenes 3D MegaDepth BlendedMVS RealEstate10k
Benchmarks: CO3D Habitat RealEstate10k
Metrics: camera rotation accuracy camera center accuracy Chamfer Distance
Baselines: COLMAP PoseDiffusion RelPose++ RayDiffusion DUSt3R DUSt3R-CO3D RD+MoGe
Builds on: RayDiffusion DUSt3R DiT DINOv2 DPT
Closest papers by meaning
- Coherent 3D Scene Diffusion From a Single RGB Image (NeurIPS 2024 · similarity 0.89)
- Cameras as Rays: Pose Estimation via Ray Diffusion (ICLR 2024 · similarity 0.88)
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.88)
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.88)
- Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.88)
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.