Research map / 3D vision / Human pose estimation
Learning Structure-from-Motion with Graph Attention Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a permutation-equivariant graph attention network (GATv2-based) that takes 2D point tracks and directly regresses camera poses and 3D points for initialization-free SfM on unseen scenes, refined with bundle adjustment.
Problem
Initialization-free Structure-from-Motion: conventional pipelines need a sequence of sub-problems to initialize bundle adjustment, and prior learned SfM methods either need scene-specific fine-tuning or generalize poorly to unseen scenes.
Approach
Replace the linear permutation-equivariant layers of DPESFM with graph cross-attention layers (GATv2) over a bipartite projection/view/scene-point/global feature graph, so the network learns SfM-specific primitives and generalizes to new sequences without fine-tuning; adds rotation-based data augmentation and artificial outlier injection.
Key results
- With augmentation and BA, average reprojection error 0.85 px vs COLMAP 0.66 px and translation error 0.07 m vs 0.03 m on 10 novel test scenes.
- Inference without BA yields average reprojection error 80.69 px with augmentation (DPESFM 436.51 px), and inference + BA is about 5-10x faster than COLMAP (e.g., 0.24 s inference, 45.5 s BA vs 286 s COLMAP on Alcatraz Courtyard).
- With outlier injection in training, uncorrupted reprojection error on average 51.91 px vs DPESFM 210.53 px.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add random rotational camera perturbation data augmentation during training (ablation) | GraphAttentionSfM · Olsson test scenes, inference + BA | average reprojection error (px): 1.92 -> 0.85 | Helped |
| Add random rotational camera perturbation data augmentation during training (ablation) | GraphAttentionSfM · Olsson test scenes, inference | average reprojection error (px): 80.66 -> 80.69 | No clear effect |
| Train with artificial outlier injection in addition to data augmentation (ablation) | GraphAttentionSfM · Olsson test scenes, outlier-free inference | average reprojection error (px): 80.69 -> 51.91 | Helped |
| Inject 10% outliers into the test measurements (ablation) | GraphAttentionSfM trained with data augmentation and outlier injection · Olsson test scenes | average reprojection error (px): 51.91 -> 60.46 | Hurt |
| Use GraphAttentionSfM rather than DPESFM | Euclidean reconstruction with augmentation + BA · Olsson test scenes | average reprojection error (px): 4.86 -> 0.85 | Helped |
| Use GraphAttentionSfM rather than DPESFM | Euclidean reconstruction with augmentation + BA · Olsson test scenes | average translation error (m): 2.38 -> 0.07 | Helped |
| Use GraphAttentionSfM rather than DPESFM | Euclidean reconstruction with augmentation + BA · Olsson test scenes | average rotation error (deg): 9.406 -> 0.144 | Helped |
| Use GraphAttentionSfM rather than Colmap | Euclidean reconstruction with augmentation + BA · Olsson test scenes | average reprojection error (px): 0.66 -> 0.85 | Hurt |
| Use GraphAttentionSfM rather than Colmap | Euclidean reconstruction with augmentation + BA · Olsson test scenes | average translation error (m): 0.03 -> 0.07 | Hurt |
| Use GraphAttentionSfM rather than Colmap | Euclidean reconstruction with augmentation + BA · Olsson test scenes | average rotation error (deg): 0.074 -> 0.144 | Hurt |
| Train with artificial outlier injection in addition to data augmentation (ablation) | GraphAttentionSfM · Olsson test scenes, 10% outlier-injected inference | average reprojection error (px): 51.91 -> 60.46 | Hurt |
Surprising findings
- Without data augmentation, the proposed model and DPESFM show no clear difference at inference, and neither reconstruction is good.
- The authors advocate against per-scene fine-tuning, noting it is costly relative to COLMAP.
Limitations
- Generalization to unseen scenes remains limited; training on larger datasets is needed.
- Still depends on bundle adjustment for accurate reconstruction, which is the computational bottleneck.
- Absolute pose prediction is weak given the similarity-transform ambiguity.
Experimental setup
Datasets: Olsson's dataset VGG dataset
Benchmarks: Olsson's dataset
Metrics: reprojection error rotation error translation error runtime
Baselines: DPESFM COLMAP
Builds on: GATv2 DPESFM Bundle Adjustment
Closest papers by meaning
- Global-Aware Edge Prioritization for Pose Graph Initialization (CVPR 2026 · similarity 0.89)
- VGGSfM: Visual Geometry Grounded Deep Structure From Motion (CVPR 2024 · similarity 0.87)
- Learning to Filter Outlier Edges in Global SfM (CVPR 2025 · similarity 0.87)
- Light3R-SfM: Towards Feed-forward Structure-from-Motion (CVPR 2025 · similarity 0.87)
- BA-Net: Dense Bundle Adjustment Networks (ICLR 2019 · similarity 0.87)
- HEIR: Learning Graph-Based Motion Hierarchies (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.