Research map / Video / Video generation
DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts Stable Video Diffusion into a video-to-depth diffusion model via a three-stage training curriculum on realistic and synthetic paired video-depth data, plus a segment-wise latent-stitching inference scheme for arbitrarily long videos.
Problem
Video depth estimation for open-world videos that are diverse in content, motion, camera movement and length, without camera poses or optical flow; existing feed-forward and test-time optimization methods flicker or need poses.
Approach
Repurpose a pretrained image-to-video diffusion model (SVD) as a conditional video-to-depth generator, training spatial and temporal layers progressively on a large binocular-derived realistic dataset and a small synthetic dataset to support variable lengths up to 110 frames, and stitch overlapping latent segments at inference for long videos.</120>
Key results
- On Sintel, DepthCrafter reaches AbsRel 0.270 and delta1 0.697, versus 0.325 / 0.564 for Depth-Anything, the best baseline.
- On KITTI (110 frames), AbsRel 0.104 vs 0.140 for Depth-Anything-V2, a 25.7% relative improvement.
- Three-stage training improves Sintel AbsRel from 0.322 (stage 1) to 0.316 (stage 2) to 0.270 (stage 3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DepthCrafter instead of Depth-Anything | open-world video depth estimation · Sintel (~50 frames) | AbsRel: 0.325 -> 0.270 | Helped |
| Use DepthCrafter instead of Depth-Anything | open-world video depth estimation · Sintel (~50 frames) | δ1: 0.564 -> 0.697 | Helped |
| Use DepthCrafter instead of Depth-Anything | open-world video depth estimation · ScanNet (90 frames) | AbsRel: 0.130 -> 0.123 | Helped |
| Use DepthCrafter instead of Depth-Anything | open-world video depth estimation · ScanNet (90 frames) | δ1: 0.838 -> 0.856 | Helped |
| Use DepthCrafter instead of Depth-Anything-V2 | open-world video depth estimation · KITTI (110 frames) | AbsRel: 0.140 -> 0.104 | Helped |
| Use DepthCrafter instead of Depth-Anything-V2 | open-world video depth estimation · KITTI (110 frames) | δ1: 0.804 -> 0.896 | Helped |
| Use DepthCrafter instead of Depth-Anything | open-world video depth estimation · Bonn (110 frames) | AbsRel: 0.078 -> 0.071 | Helped |
| Use DepthCrafter instead of Depth-Anything | open-world video depth estimation · Bonn (110 frames) | δ1: 0.939 -> 0.972 | Helped |
| Use DepthCrafter instead of Depth-Anything | single-image depth estimation · NYU-v2 (1 frame) | AbsRel: 0.042 -> 0.072 | Hurt |
| Use DepthCrafter instead of Depth-Anything | single-image depth estimation · NYU-v2 (1 frame) | δ1: 0.981 -> 0.948 | Hurt |
| Add Stage 2 temporal-layer fine-tuning to Stage 1 (ablation) | DepthCrafter · Sintel | AbsRel: 0.322 -> 0.316 | Helped |
| Add Stage 2 temporal-layer fine-tuning to Stage 1 (ablation) | DepthCrafter · Sintel | δ1: 0.626 -> 0.675 | Helped |
Surprising findings
- On NYU-v2 single-image depth, DepthCrafter (AbsRel 0.072) is worse than Depth-Anything (0.042) and Depth-Anything-V2 (0.043) in AbsRel, though its delta1 is competitive.
Limitations
- High computation and memory cost due to large model and iterative denoising.
- Slower than feed-forward Depth-Anything (V2) per frame.
- Training used a small synthetic dataset (~3K sequences) and a large realistic dataset generated by a stereo method.
Experimental setup
Datasets: BiDAStereo-labeled binocular videos (realistic, ~200K) DynamicReplica MatrixCity Sintel ScanNet v2 Bonn KITTI NYU-v2 DAVIS
Benchmarks: Sintel ScanNet v2 KITTI Bonn NYU-v2
Metrics: AbsRel delta1
Baselines: Marigold Depth-Anything Depth-Anything-V2 NVDS ChronoDepth
Builds on: Stable Video Diffusion Marigold EDM Latent Diffusion Models
Closest papers by meaning
- Depth Any Video with Scalable Synthetic Data (ICLR 2025 · similarity 0.90)
- Video Depth Anything: Consistent Depth Estimation for Super-Long Videos (CVPR 2025 · similarity 0.90)
- WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perception (NeurIPS 2025 · similarity 0.87)
- GeoVideo: Introducing Geometric Regularization into Video Generation Model (NeurIPS 2025 · similarity 0.86)
- Learning Temporally Consistent Video Depth from Video Diffusion Priors (CVPR 2025 · similarity 0.86)
- Seurat: From Moving Points to Depth (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.