Research map / Video / Video segmentation and tracking
Video Depth Anything: Consistent Depth Estimation for Super-Long Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Video Depth Anything by adding a temporal-attention spatiotemporal head to a frozen Depth Anything V2 encoder, trained with a temporal gradient matching loss and a key-frame inference strategy for minutes-long consistent video depth.
Problem
Monocular depth foundation models like Depth Anything V2 flicker on video, and existing video depth methods are limited to short clips (under ~10 seconds), are slow, or depend on optical flow or camera poses.
Approach
Replace the DPT head with a spatiotemporal head containing temporal self-attention layers on a frozen DAv2 encoder, supervise with a flow-free temporal gradient matching loss on depth changes at the same pixel, and run long-video inference with overlap interpolation plus key-frame referencing to limit scale drift.
Key results
- On KITTI, ScanNet, and Bonn, VDA-L improves delta1 by roughly 10% over prior video depth methods (e.g., KITTI delta1 0.946 vs 0.753 for DepthCrafter per Table 1 setting).
- VDA-L latency is 67 ms per frame on A100 versus 910 ms for DepthCrafter; VDA-S runs at 9.1 ms per frame.
- Temporal TAE on Scannet (170 frames) drops to 0.570 for VDA-L versus 0.639 for DepthCrafter and 1.140 for DAv2-L.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the image-depth head with a spatiotemporal head and train with temporal consistency | Depth Anything V2-L · KITTI | AbsRel: 0.137 -> 0.083 | Helped |
| Replace the image-depth head with a spatiotemporal head and train with temporal consistency | Depth Anything V2-L · Scannet | δ1: 0.768 -> 0.926 | Helped |
| Replace the image-depth head with a spatiotemporal head and train with temporal consistency | Depth Anything V2-L · Bonn | AbsRel: 0.127 -> 0.071 | Helped |
| Replace the image-depth head with a spatiotemporal head and train with temporal consistency | Depth Anything V2-L · NYUv2 | δ1: 0.928 -> 0.971 | Helped |
| Replace the image-depth head with a spatiotemporal head and train with temporal consistency | Depth Anything V2-L · Sintel | AbsRel: 0.390 -> 0.295 | Hurt |
| Use VideoAlign+SSI instead of VideoAlign (ablation) | VDA-S · TartanAir and VKITTI | TAE: 1.326 -> 1.207 | Helped |
| Use temporal gradient matching with SSI instead of optical-flow warping with SSI (ablation) | VDA-S · TartanAir and VKITTI | TAE: 0.918 -> 0.767 | Helped |
| Use temporal gradient matching with SSI instead of stable-error matching with SSI (ablation) | VDA-S · TartanAir and VKITTI | TAE: 0.753 -> 0.767 | No clear effect |
| Add overlap interpolation and key-frame referencing instead of independent window inference (ablation) | VDA-S · Video depth evaluation datasets | TAE: 0.874 -> 0.761 | Helped |
| Increase window size from 16 to 32 with overlap interpolation and key-frame referencing (ablation) | VDA-S · Video depth evaluation datasets | TAE: 0.761 -> 0.718 | Helped |
| Increase window size from 32 to 48 with overlap interpolation and key-frame referencing (ablation) | VDA-S · Video depth evaluation datasets | TAE: 0.718 -> 0.732 | Hurt |
| Add image dataset distillation to video training (ablation) | VDA-S · Image benchmarks | AbsRel: 0.180 -> 0.167 | Helped |
Surprising findings
- DepthCrafter scores better than VDA on Sintel short videos, which the authors attribute to missing movie data in training.
- The overlap alignment (OA) inference variant matches or beats the proposed OI+KR on the 500-frame benchmarks but accumulates scale drift on a 4-minute video.
Limitations
- Sintel performance trails DepthCrafter, attributed to lack of movie-style training data.
- Long-video scale drift is only evaluated qualitatively on a self-captured video, since benchmark videos are at most 500 frames.
Experimental setup
Datasets: TartanAir VKITTI DAv2 synthetic video depth data 0.62M unlabeled images
Benchmarks: KITTI ScanNet Bonn NYUv2 Sintel DAVIS ETH3D DIODE
Metrics: AbsRel delta1 TAE latency
Baselines: Depth Anything V2 NVDS NVDS + DAv2 ChronoDepth DepthCrafter DepthAnyVideo
Builds on: Depth Anything V2 DPT NVDS MiDaS
Closest papers by meaning
- Depth Any Video with Scalable Synthetic Data (ICLR 2025 · similarity 0.92)
- Buffer Anytime: Zero-Shot Video Depth and Normal from Image Priors (CVPR 2025 · similarity 0.91)
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (CVPR 2024 · similarity 0.90)
- Depth Anything V2 (NeurIPS 2024 · similarity 0.90)
- DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos (CVPR 2025 · similarity 0.90)
- Seurat: From Moving Points to Depth (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.