Research map / 3D vision / Depth and stereo
MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes DUSt3R's pointmap regression on a small mix of dynamic synthetic and driving datasets, then adds video-specific global optimization (smoothness and flow losses) to get video depth and camera pose from monocular dynamic videos.
Problem
Estimating geometry (depth and camera poses) from monocular videos of dynamic scenes, where moving objects violate static-scene assumptions and multi-stage pipelines are slow and error-prone; dynamic posed video data with depth is scarce.
Approach
Represent each timestep with a per-frame pointmap in DUSt3R's framework and adapt DUSt3R to dynamic scenes by fine-tuning only its decoder and heads on a small mixture of dynamic posed datasets with stride-based pair sampling, then do sliding-window global optimization with camera smoothness and flow-consistency losses.
Key results
- Joint video depth and pose: MonST3R achieves Sintel video depth Abs Rel 0.335 (per-sequence scale and shift) and best Bonn (0.063) and KITTI (0.104) among compared methods, outperforming CasualSAM and Robust-CVD.
- Camera pose on Sintel: ATE 0.108 for MonST3R vs 0.141 for CasualSAM and 0.360 for Robust-CVD, without ground-truth intrinsics.
- Global optimization runs in about 1 minute for a 60-frame video on one RTX 6000; a 60-frame inference takes around 30s; fully feed-forward mode reaches about 40 FPS on RTX 4090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune DUSt3R on all four training datasets (ablation) | DUSt3R · Sintel camera pose | ATE: 0.354 -> 0.108 | Helped |
| Add TartanAir to PointOdyssey fine-tuning (ablation) | MonST3R · Sintel camera pose | ATE: 0.220 -> 0.158 | Helped |
| Add Spring to PointOdyssey and TartanAir fine-tuning (ablation) | MonST3R · Sintel camera pose | ATE: 0.158 -> 0.121 | Helped |
| Add Waymo to PointOdyssey, TartanAir, and Spring fine-tuning (ablation) | MonST3R · Sintel camera pose | ATE: 0.121 -> 0.108 | Helped |
| Fine-tune only the decoder and head instead of the full model (ablation) | DUSt3R · Sintel camera pose | ATE: 0.181 -> 0.108 | Helped |
| Fine-tune the decoder and head rather than the head only (ablation) | DUSt3R · Sintel camera pose | ATE: 0.185 -> 0.108 | Helped |
| Add flow loss (ablation) | MonST3R global optimization · Sintel camera pose | ATE: 0.140 -> 0.108 | Helped |
| Add static region mask (ablation) | MonST3R global optimization · Sintel camera pose | ATE: 0.132 -> 0.108 | Helped |
| Add camera trajectory smoothness loss (ablation) | MonST3R global optimization · Sintel camera pose | ATE: 0.127 -> 0.108 | Helped |
| Use MonST3R instead of DepthCrafter | Video depth estimation · Bonn, per-sequence scale & shift | Abs Rel: 0.075 -> 0.063 | Helped |
| Use MonST3R instead of DepthCrafter | Video depth estimation · Sintel, per-sequence scale & shift | Abs Rel: 0.292 -> 0.335 | Hurt |
| Use MonST3R instead of DepthCrafter | Video depth estimation · KITTI, per-sequence scale & shift | Abs Rel: 0.110 -> 0.104 | Helped |
Surprising findings
- Fine-tuning DUSt3R on a small dynamic dataset mixture enables handling of motion without any explicit motion representation.
- Simply masking dynamic objects in DUSt3R (with ground-truth masks) degrades pose estimation, attributed to out-of-distribution black pixels and mask tokens.
Limitations
- Small sliding temporal window makes global alignment vulnerable to long-term occlusion.
- Struggles with out-of-distribution inputs such as open fields; depends on training data coverage.
- Handling dynamic camera intrinsics requires careful hyperparameter tuning or manual constraints in practice.
Experimental setup
Datasets: PointOdyssey TartanAir Spring Waymo Perception Sintel Bonn KITTI NYU-v2 TUM-dynamics ScanNet DAVIS
Benchmarks: Sintel Bonn KITTI NYU-v2 TUM-dynamics ScanNet
Metrics: Abs Rel delta<1.25 ATE RPE trans RPE rot
Baselines: DepthCrafter Depth-Anything-V2 Marigold NVDS ChronoDepth CasualSAM Robust-CVD DROID-SLAM DPVO LEAP-VO ParticleSfM DUSt3R
Builds on: DUSt3R CroCo RAFT
Closest papers by meaning
- Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos (CVPR 2025 · similarity 0.90)
- DUSt3R: Geometric 3D Vision Made Easy (CVPR 2024 · similarity 0.89)
- Align3R: Aligned Monocular Depth Estimation for Dynamic Videos (CVPR 2025 · similarity 0.89)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.88)
- MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds (CVPR 2025 · similarity 0.88)
- Layered Motion Fusion: Lifting Motion Segmentation to 3D in Egocentric Videos (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.