Research map / 3D vision / Depth and stereo
VGGSfM: Visual Geometry Grounded Deep Structure From Motion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VGGSfM, a fully differentiable end-to-end structure-from-motion pipeline that predicts tracks with a deep point tracker, jointly estimates all cameras with Transformers, and refines via a differentiable bundle adjustment layer.
Problem
Classical incremental structure-from-motion is non-differentiable and built from hand-engineered stages (keypoint matching, chaining, incremental registration), which complicates end-to-end learning and limits how learned components can be combined.
Approach
Replace pairwise matching and incremental registration with a feed-forward multi-frame point tracker (coarse-to-fine, no temporal assumption), jointly predict all cameras and an initial point cloud with Transformers, and refine with a differentiable Levenberg-Marquardt bundle adjustment (Theseus), training the whole pipeline end to end.
Key results
- On Co3D, VGGSfM reaches AUC@30 of 74.0 and RRE@15 of 92.1, about +9 points over the prior deep method PoseDiffusion (AUC@30 66.5).
- On IMC Phototourism, VGGSfM reaches AUC@10 of 73.92, outperforming PixSfM (SP+SG) at 70.47 and DFSfM (LoFTR) at 72.19.
- On ETH3D at 5cm, completeness is 33.96% versus 29.54% for the best prior work (DFSfM LoFTR).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train all components jointly end-to-end (ablation) | VGGSfM · Co3D | AUC@30°: 70.7% to 74.0% | Helped |
| train all components jointly end-to-end (ablation) | VGGSfM · IMC Phototourism | AUC@10°: 68.35% to 73.92% | Helped |
| use learned tracks instead of SP+SG tracks (ablation) | VGGSfM · IMC Phototourism | AUC@10°: 68.78 to 73.92 | Helped |
| use learned tracks with PixSfM instead of SP+SG matching (ablation) | PixSfM · IMC Phototourism | AUC@10°: 70.47 to 70.62 | No clear effect |
| use the VGGSfM camera initializer instead of PoseDiffusion (ablation) | VGGSfM · IMC Phototourism | AUC@10°: 62.18 to 73.92 | Helped |
| use the learned triangulator instead of DLT (ablation) | VGGSfM · IMC Phototourism | AUC@10°: 66.37 to 73.92 | Helped |
| add fine tracking in the coarse-to-fine tracker (ablation) | VGGSfM · IMC Phototourism | AUC@10°: 62.30% to 73.92% | Helped |
Surprising findings
- Tracks from the learned tracker outperform SuperPoint+SuperGlue matches fed to PixSfM on IMC (AUC@10 70.62 vs 70.47), while feeding SP+SG tracks into VGGSfM drops performance.
Limitations
- Currently cannot process thousands of images as traditional SfM frameworks can.
- Does not yet compete with established pipelines in all application domains.
- Initial camera estimator and triangulator rely on tracker-derived features and the query-frame iteration procedure.
Experimental setup
Datasets: Kubric Co3Dv2 MegaDepth IMC Phototourism ETH3D
Benchmarks: Co3Dv2 IMC Phototourism ETH3D
Metrics: AUC@30 RRE@15 RTE@15 AUC@3 AUC@5 AUC@10 accuracy completeness
Baselines: COLMAP (SP + SG) PixSfM RelPose PoseReg RelPose++ PoseDiffusion DeepSFM DFSfM PatchFlow
Builds on: PIPs CoTracker TAPIR PoseDiffusion Theseus PixSfM
Closest papers by meaning
- VGGT: Visual Geometry Grounded Transformer (CVPR 2025 · similarity 0.89)
- Detector-Free Structure from Motion (CVPR 2024 · similarity 0.88)
- MP-SfM: Monocular Surface Priors for Robust Structure-from-Motion (CVPR 2025 · similarity 0.88)
- Light3R-SfM: Towards Feed-forward Structure-from-Motion (CVPR 2025 · similarity 0.88)
- ColabSfM: Collaborative Structure-from-Motion by Point Cloud Registration (CVPR 2025 · similarity 0.88)
- Deep Non-Rigid Structure-from-Motion Revisited: Canonicalization and Sequence Modeling (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.