Research map / Video / Video segmentation and tracking
Depth Any Video with Scalable Synthetic Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 40K-clip synthetic video depth dataset (DA-V) from game engines and fine-tunes Stable Video Diffusion into a generative video depth model with flow matching, mixed-duration training, and keyframe-based long-video interpolation.
Problem
Video depth estimation lacks large-scale, diverse, temporally consistent ground-truth depth data, and existing generative depth models struggle with temporal consistency and long or high-resolution videos.
Approach
Collect synthetic video depth from real-time game engines via ReShade/OBS, then adapt Stable Video Diffusion as a conditional flow-matching depth denoiser trained with RoPE-based frame dropout and resolution-grouped video packing, plus a keyframe-conditioned interpolation module for long videos.
Key results
- On zero-shot single-frame benchmarks, Depth Any Video reports AbsRel 5.1 / delta1 97.0 on NYUv2 and 4.7 / 97.9 on ETH3D, beating GeoWizard (5.2 / 96.6 and 6.4 / 96.1) among generative models.
- Flow matching with 1 vs 25 EDM steps gives a 6.5x inference speedup and improves AbsRel by 0.6 over the EDM scheduler in the ablation.
- Mixed-duration training saves 33% training time and raises minimum GPU memory utilization from 23% to 63% relative to the fixed-length setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the Stable Video Diffusion generative visual prior (ablation) | Depth Any Video · Four-dataset average | AbsRel: 21.0 -> 7.5 | Helped |
| Add the Stable Video Diffusion generative visual prior (ablation) | Depth Any Video · Four-dataset average | δ1 accuracy: 65.1 -> 93.8 | Helped |
| Replace the EDM scheduler with conditional flow matching (ablation) | Depth Any Video with generative visual prior · Four-dataset average | AbsRel: 7.5 -> 6.9; 0.6 improvement | Helped |
| Replace the EDM scheduler with conditional flow matching (ablation) | Depth Any Video with generative visual prior · Four-dataset average | δ1 accuracy: 93.8 -> 94.9; 1.1 improvement | Helped |
| Replace the EDM scheduler with conditional flow matching (ablation) | Depth Any Video with generative visual prior · Ablation inference | runtime (s): 2.4 -> 0.37; 6.5× acceleration | Helped |
| Add the DA-V synthetic dataset (ablation) | Depth Any Video with generative prior and flow matching · Four-dataset average | AbsRel: 6.9 -> 6.5; 0.4 improvement | Helped |
| Add the DA-V synthetic dataset (ablation) | Depth Any Video with generative prior and flow matching · Four-dataset average | δ1 accuracy: 94.9 -> 95.6; 0.7 improvement | Helped |
| Add mixed-duration training (ablation) | Depth Any Video with generative prior, flow matching, and synthetic data · Four-dataset average | AbsRel: 6.5 -> 6.4 | Helped |
| Add mixed-duration training (ablation) | Depth Any Video with generative prior, flow matching, and synthetic data · Four-dataset average | δ1 accuracy: 95.6 -> 95.8 | Helped |
| Add mixed-duration training (ablation) | Depth Any Video with generative prior, flow matching, and synthetic data · Ablation training | training time (hours): 16 -> 12; 33% of training time | Helped |
| Add mixed-duration training (ablation) | Depth Any Video with generative prior, flow matching, and synthetic data · Ablation training | GPU memory utilization (%): 23 -> 63; 40% | Helped |
Surprising findings
- The paper reports that 2D (spatially-only) VAEs reconstruct depth better than 3D temporal VAEs, which blur fast motion.
Limitations
- Struggles with mirror-like reflections on water surfaces.
- Difficulties with extremely long videos.
- Relies on ensembling and multiple denoising steps for best benchmark numbers.
Experimental setup
Datasets: DA-V Hypersim Virtual KITTI 2 NYUv2 ScanNet ETH3D KITTI ScanNet++
Benchmarks: NYUv2 KITTI ETH3D ScanNet ScanNet++
Metrics: AbsRel delta-1 accuracy temporal alignment error (TAE) runtime
Baselines: Depth Anything Marigold GeoWizard DepthFM Metric3D Omnidata DPT ChronoDepth DepthCrafter Depth Anything V2
Builds on: Stable Video Diffusion Marigold conditional flow matching rotary position encoding Depth Anything
Closest papers by meaning
- Video Depth Anything: Consistent Depth Estimation for Super-Long Videos (CVPR 2025 · similarity 0.92)
- DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos (CVPR 2025 · similarity 0.90)
- Learning Temporally Consistent Video Depth from Video Diffusion Priors (CVPR 2025 · similarity 0.89)
- DepthFM: Fast Generative Monocular Depth Estimation with Flow Matching (AAAI 2025 · similarity 0.89)
- Depth Anything V2 (NeurIPS 2024 · similarity 0.88)
- Seurat: From Moving Points to Depth (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.