Research map / Video / Video segmentation and tracking
Align3R: Aligned Monocular Depth Estimation for Dynamic Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Align3R fine-tunes DUSt3R with features from monocular depth maps (Depth Pro or Depth Anything V2) injected via a zero-convolution point-map ViT, then uses hierarchical global alignment to produce temporally consistent video depth and camera poses from monocular videos.
Problem
Monocular depth estimators produce per-frame depth with inconsistent scale, causing flicker in video depth; video diffusion methods are training-expensive, clip-length limited, and give scale-invariant depth without camera poses.
Approach
Inject unprojected monocular depth point maps into DUSt3R's decoder through a new ViT encoder with zero convolutions (ControlNet-style) and fine-tune the decoder on dynamic videos, so pairwise point maps are aligned across frames and globally optimized into consistent depth and poses.
Key results
- Video depth on Sintel: Abs Rel 0.078 (Depth Anything V2 variant) vs. 0.089 for MonST3R, and 0.078 vs. 0.335 for MonST3R on the hard indoor/outdoor split.
- Camera pose on TUM dynamics1 ATE 0.011 vs. 0.020 for MonST3R.
- Ablation: concatenating depth with RGB degrades Sintel Abs Rel to 0.399 vs. 0.263 with the proposed ViT encoder injection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Align3R with Depth Anything V2 instead of MonST3R | video depth estimation · Sintel | Abs Rel: 0.335 -> 0.253 | Helped |
| Use Align3R with Depth Anything V2 instead of MonST3R | video depth estimation · PointOdyssey val | Abs Rel: 0.089 -> 0.078 | Helped |
| Use Align3R with Depth Anything V2 instead of MonST3R | video depth estimation · FlyingThings3D test | Abs Rel: 0.132 -> 0.106 | Helped |
| Use Align3R with Depth Anything V2 instead of MonST3R | video depth estimation · Bonn 5 scenes | Abs Rel: 0.082 -> 0.075 | Helped |
| Use Align3R with Depth Anything V2 instead of MonST3R | video depth estimation · TUM dynamics | Abs Rel: 0.140 -> 0.109 | Helped |
| Fine-tune the whole DUSt3R model rather than only the decoder (ablation) | Align3R · Sintel | Abs Rel: 0.306 -> 0.310 | Hurt |
| Fine-tune only the last 4 decoder layers rather than the whole decoder (ablation) | Align3R · Sintel | Abs Rel: 0.306 -> 0.319 | Hurt |
| Add the estimated monocular depth input (ablation) | Align3R · Sintel | Abs Rel: 0.306 -> 0.263 | Helped |
| Concatenate the depth map with the input RGB images (ablation) | Align3R · Sintel | Abs Rel: 0.263 -> 0.399 | Hurt |
| Use hierarchical optimization instead of the base optimization (ablation) | Align3R · Bonn | Abs Rel: 0.054 -> 0.056 | Hurt |
Surprising findings
- Naively concatenating monocular depth with RGB inputs hurts performance versus no depth input, breaking the pretrained DUSt3R encoder.
- On simpler indoor Bonn/TUM scenes, the raw Depth Pro predictions are already consistent and the method does not clearly beat them, sometimes losing detail in global alignment.
Limitations
- Relies on external monocular depth estimator quality; fine-tuning uses only synthetic datasets.
- Gains are marginal on simple indoor scenes where raw monocular depth is already consistent.
- Global alignment still requires optimization at inference, and ATE is slightly worse than MonST3R on Sintel.
Experimental setup
Datasets: SceneFlow VKITTI TartanAir Spring PointOdyssey
Benchmarks: Sintel PointOdyssey FlyingThings3D Bonn TUM dynamics ScanNetV2 NYU-v2 KITTI
Metrics: Abs Rel delta<1.25 ATE RTE RRE RPE
Baselines: Depth Anything V2 Depth Pro ChronoDepth DepthCrafter DUSt3R MonST3R DROID-SLAM DPVO COLMAP Robust-CVD CasualSAM
Builds on: DUSt3R ControlNet Depth Pro Depth Anything V2 MonST3R
Closest papers by meaning
- MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (ICLR 2025 · similarity 0.89)
- Video Depth Anything: Consistent Depth Estimation for Super-Long Videos (CVPR 2025 · similarity 0.88)
- Depth2Pose: A Pose-Based Benchmark for Monocular Depth Estimation without Ground-Truth Depth (NeurIPS 2026 · similarity 0.88)
- DUSt3R: Geometric 3D Vision Made Easy (CVPR 2024 · similarity 0.88)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.87)
- Seurat: From Moving Points to Depth (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.