Research map / 3D vision / Depth and stereo
RESfM: Robust Deep Equivariant Structure from Motion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends the equivariant deep SfM network ESFM with a sets-of-sets inlier/outlier classifier and a robust bundle adjustment so it works on outlier-contaminated point tracks from internet photo collections.
Problem
Deep multiview structure from motion (camera pose and 3D structure recovery) from point tracks; prior equivariant deep methods assume nearly outlier-free tracks, failing on realistic internet photo collections with many outliers.
Approach
Adds a permutation-equivariant (sets-of-sets) multiview inlier/outlier classification head trained with BCE on COLMAP-derived labels, followed by fine-tuning on the test scene and a robust BA procedure that removes high-error points and disconnected views.
Key results
- On MegaDepth test scenes, mean rotation error 1.29 deg and translation 0.169 vs ESFM 6.77/0.780 and GASFM 4.53/0.630 (mean over scenes).
- Inlier/outlier classifier reduces predicted outlier ratio from 31.9% input to 19.0% and 11.8% after robust BA on average (1DSfM subset of scenes in ablation table).
- Classification F-score averages 54.1 with the sets-of-sets architecture vs 48.8 for PointNet on Alamo (mean F-score 54.1 vs 44.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the equivariant inlier/outlier classifier rather than using PointNet (ablation) | RESfM · 1DSFM scenes (mean) | outlier F-score: 44.9 -> 54.1 | Helped |
| add robust bundle adjustment instead of standard bundle adjustment (ablation) | RESfM on Alamo · 1DSFM (Alamo) | rotation error: 10.18 -> 3.66 | Helped |
| add robust bundle adjustment instead of standard bundle adjustment (ablation) | RESfM on Alamo · 1DSFM (Alamo) | translation error: 1.970 -> 0.515 | Helped |
| train the feature encoder end-to-end rather than freezing it (ablation) | RESfM on Alamo · 1DSFM (Alamo) | rotation error: 3.79 -> 3.66 | No clear effect |
| remove the feature encoder and use the ESFM configuration (ablation) | RESfM on Alamo · 1DSFM (Alamo) | rotation error: 3.66 -> 4.53 | Hurt |
| use RESfM rather than ESFM on contaminated tracks | SfM methods · MegaDepth test (mean) | rotation error: 6.77 -> 1.29 | Helped |
| use RESfM rather than GASFM on contaminated tracks | SfM methods · MegaDepth test (mean) | translation error: 0.630 -> 0.169 | Helped |
| use RESfM rather than Theia | SfM methods · MegaDepth test (mean) | rotation error: 2.42 -> 1.29 | Hurt |
| use RESfM rather than Theia | SfM methods · Strecha (mean) | runtime: 1.4 -> 5.2 | Hurt |
Surprising findings
- Classical GLOMAP and Theia often achieve lower rotation/translation error than the proposed method on several scenes, and the method registers fewer cameras in some cases (e.g., Tower of London 94 of 467 images).
Limitations
- Excessive removal of predicted outliers can leave the view graph disconnected, so only a subset of cameras is used for reconstruction.
- Slower than Theia and fewer registered images than classical methods on some scenes.
- Test on large collections uses 300-image subsamples of MegaDepth Group 2 scenes.
Experimental setup
Datasets: MegaDepth 1DSfM Strecha BlendedMVS
Benchmarks: MegaDepth 1DSfM Strecha BlendedMVS
Metrics: rotation error (degrees) translation error number of registered cameras runtime precision recall F-score
Baselines: ESFM GASFM Theia GLOMAP COLMAP VGGSfM MASt3R
Builds on: ESFM GASFM Ceres Solver bundle adjustment SIFT COLMAP
Closest papers by meaning
- VGGSfM: Visual Geometry Grounded Deep Structure From Motion (CVPR 2024 · similarity 0.87)
- Detector-Free Structure from Motion (CVPR 2024 · similarity 0.85)
- MP-SfM: Monocular Surface Priors for Robust Structure-from-Motion (CVPR 2025 · similarity 0.85)
- PARSAC: Accelerating Robust Multi-Model Fitting with Parallel Sample Consensus (AAAI 2024 · similarity 0.84)
- A Regularization-Guided Equivariant Approach for Image Restoration (CVPR 2025 · similarity 0.84)
- A Subspace-Constrained Tyler's Estimator and its Applications to Structure from Motion (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.