Research map / 3D vision / Depth and stereo
MegaSaM: Accurate, Fast and Robust Structure and Motion from Casual Dynamic Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends the DROID-SLAM deep visual SLAM framework with a learned motion probability map, mono-depth initialization/regularization, uncertainty-aware global BA, and consistent video depth optimization to estimate cameras and depth from casual dynamic monocular videos.
Problem
Estimating camera poses, focal length, and dense depth from casual monocular videos of dynamic scenes with limited parallax and unknown field of view, where classical SfM/SLAM and existing learned methods fail or are slow.
Approach
Couples a differentiable deep visual SLAM bundle adjustment layer with a learned object-motion probability map that downweights dynamic pixels, mono-depth priors for disparity initialization and regularization, and an uncertainty (Hessian-diagonal) criterion that decides when to apply mono-depth regularization and whether to optimize focal length.
Key results
- On Sintel calibrated camera estimation, MegaSaM reaches ATE 0.018 vs 0.036 for CasualSAM and 0.041 for LEAP-VO, at about 1.0s per frame.
- On DyCheck calibrated, ATE 0.020 vs 0.062 for the next best (ACE-Zero), with 0.8s runtime.
- Video depth on Sintel reaches abs-rel 0.21 and delta1.25 73.1 versus 0.31 and 64.2 for CasualSAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MegaSaM instead of the strongest listed baseline for camera tracking | Sintel · Sintel, calibrated | ATE: 0.041 -> 0.018 | Helped |
| Use MegaSaM instead of the strongest listed baseline for camera tracking | DyCheck · DyCheck, calibrated | ATE: 0.062 -> 0.020 | Helped |
| Use MegaSaM instead of the strongest listed baseline for camera tracking | In-the-Wild · In-the-Wild, calibrated | ATE: 0.016 -> 0.004 | Helped |
| Add mono-depth initialization (ablation) | MegaSaM · Sintel | ATE: 0.038 -> 0.019 | Helped |
| Add object movement map prediction (ablation) | MegaSaM · Sintel | ATE: 0.032 -> 0.019 | Helped |
| Use two-stage training rather than direct training on dynamic videos (ablation) | MegaSaM · Sintel | ATE: 0.035 -> 0.019 | Helped |
| Add uncertainty-aware global BA (ablation) | MegaSaM · Sintel | ATE: 0.033 -> 0.019 | Helped |
| Jointly refine camera poses during depth optimization (ablation) | MegaSaM · Sintel | ATE: 0.019 -> 0.041 | Hurt |
| Jointly refine camera poses during depth optimization (ablation) | MegaSaM · Sintel | abs-rel: 0.21 -> 0.23 | Hurt |
| Replace the proposed depth-prior loss with CasualSAM's original mono-depth prior loss (ablation) | MegaSaM · Sintel | abs-rel: 0.21 -> 0.36 | Hurt |
| Replace the proposed depth-prior loss with CasualSAM's original mono-depth prior loss (ablation) | MegaSaM · Sintel | δ1.25: 73.1 -> 72.5 | Hurt |
Surprising findings
- Vanilla direct training on dynamic videos produces unstable training behavior through differentiable BA, motivating the two-stage scheme.
- Reprojection-only global BA can degenerate under rotation-dominant near-stationary cameras, requiring mono-depth regularization.
Limitations
- Fails when moving objects dominate the image or there is little trackable content.
- Cannot handle videos with varying focal lengths or strong radial distortion.
- Camera tracking training takes about 4 days on 8 A100s.
Experimental setup
Datasets: TartanAir Kubric Sintel DyCheck DAVIS DynIBaR in-the-wild videos
Benchmarks: MPI Sintel DyCheck In-the-Wild (DynIBaR videos)
Metrics: ATE RTE RRE abs-rel log-rmse delta1.25
Baselines: CasualSAM LEAP-VO ACE-Zero Particle-SfM RoDynRF MonST3R DepthCrafter DA-v2
Builds on: DROID-SLAM CasualSAM DepthAnything UniDepth
Closest papers by meaning
- Align3R: Aligned Monocular Depth Estimation for Dynamic Videos (CVPR 2025 · similarity 0.87)
- SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videos (CVPR 2025 · similarity 0.87)
- Depth2Pose: A Pose-Based Benchmark for Monocular Depth Estimation without Ground-Truth Depth (NeurIPS 2026 · similarity 0.86)
- ProDyG: Progressive Dynamic Scene Reconstruction via Gaussian Splatting from Monocular Videos (NeurIPS 2025 · similarity 0.86)
- Multi-Session SLAM with Differentiable Wide-Baseline Pose Optimization (CVPR 2024 · similarity 0.86)
- Dynamic Camera Poses and Where to Find Them (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.