Research map / Video / Video segmentation and tracking
Video Depth without Video Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Turns a single-image latent diffusion depth model (Marigold) into a video depth estimator by fine-tuning it on short multi-frame snippets, sampling snippets at multiple dilation rates, and globally co-aligning them with robust optimization.
Problem
Monocular video depth estimation needs temporally consistent depth per frame, but frame-by-frame single-image estimators flicker and drift, while video diffusion models are expensive and limited to short fixed-length clips.
Approach
Extend a single-image LDM (Marigold) with flattened cross-frame self-attention over short snippets sampled at varying frame spacings, then recover a consistent long depth video by optimizing per-snippet scale and shift (co-alignment), with an optional partial-diffusion refinement.
Key results
- RollingDepth reaches 9.6% AbsRel on PointOdyssey (250 frames) versus 14.4% for Depth Anything V2 and 36.3% for DepthCrafter.
- On ScanNet (90 frames), RollingDepth achieves 9.3% AbsRel and 91.6 delta1, the best among compared methods.
- Ablation shows co-alignment drives most of the gain (PointOdyssey AbsRel 13.0 to 10.2 with co-alignment), while refinement adds only marginal metric change.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RollingDepth instead of DepthAnythingv2 | Monocular video depth estimation · PointOdyssey (250) | Abs Rel (%): 14.4 -> 9.6 | Helped |
| use RollingDepth instead of DepthAnythingv2 | Monocular video depth estimation · PointOdyssey (250) | δ1 (%): 81.4 -> 90.5 | Helped |
| use RollingDepth instead of DepthCrafter | Monocular video depth estimation · ScanNet (90) | Abs Rel (%): 12.7 -> 9.3 | Helped |
| use RollingDepth instead of DepthCrafter | Monocular video depth estimation · Bonn (110) | Abs Rel (%): 6.6 -> 7.9 | Hurt |
| use RollingDepth instead of DepthCrafter | Monocular video depth estimation · DyDToF (200) | Abs Rel (%): 22.1 -> 17.3 | Helped |
| add dilation rate 25 to snippet sampling (ablation) | RollingDepth · PointOdyssey | Abs Rel (%): 16.7 -> 10.2 | Helped |
| add dilation rate 25 to snippet sampling (ablation) | RollingDepth · ScanNet | Abs Rel (%): 12.8 -> 10.6 | Helped |
| add dilation rate 10 to dilation rates {1, 25} (ablation) | RollingDepth · PointOdyssey | Abs Rel (%): 10.2 -> 10.2 | No clear effect |
| add depth co-alignment (ablation) | RollingDepth snippet predictions without refinement · PointOdyssey | Abs Rel (%): 13.0 -> 10.2 | Helped |
| add depth co-alignment (ablation) | RollingDepth snippet predictions without refinement · ScanNet | Abs Rel (%): 12.4 -> 9.9 | Helped |
| add depth refinement after co-alignment (ablation) | RollingDepth · PointOdyssey | δ1 (%): 89.6 -> 89.8 | No clear effect |
| add depth refinement after co-alignment (ablation) | RollingDepth · ScanNet | δ1 (%): 90.1 -> 90.2 | No clear effect |
Surprising findings
- Video diffusion-based depth models (ChronoDepth, DepthCrafter) fail to match single-frame methods on PointOdyssey, especially with sudden depth range changes.
- Refinement step gives only marginal metric gains despite visibly sharper details.
Limitations
- Refinement adds inference cost with only small metric gains.
- Depends on a per-snippet affine-invariant depth model and global co-alignment; the paper notes reliance on snippet overlap for alignment.
Experimental setup
Datasets: TartanAir Hypersim PointOdyssey ScanNet Bonn RGBD DyDToF
Benchmarks: PointOdyssey ScanNet Bonn RGBD DyDToF
Metrics: AbsRel delta1 accuracy
Baselines: Marigold Depth Anything Depth Anything V2 NVDS ChronoDepth DepthCrafter
Builds on: Marigold Stable Diffusion Hypersim TartanAir
Closest papers by meaning
- Video Depth Anything: Consistent Depth Estimation for Super-Long Videos (CVPR 2025 · similarity 0.86)
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (CVPR 2024 · similarity 0.81)
- Depth Any Video with Scalable Synthetic Data (ICLR 2025 · similarity 0.81)
- Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (ICLR 2025 · similarity 0.81)
- Buffer Anytime: Zero-Shot Video Depth and Normal from Image Priors (CVPR 2025 · similarity 0.81)
- Depth Anything V2 (NeurIPS 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.