Research map / Video / Video generation
Learning Temporally Consistent Video Depth from Video Diffusion Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Repurposes Stable Video Diffusion into a streamed video depth estimator (ChronoDepth) that shares context within and across clips via per-frame noise levels and noise-free overlapping context frames.
Problem
Streamed video depth estimation needs both per-frame accuracy and cross-frame temporal consistency, and existing single-image diffusion depth models flicker, while sliding-window and replacement-trick inference leave inconsistency across clips.
Approach
Fine-tune SVD as a conditional denoiser that samples independent noise levels per frame during training, and at inference initializes overlapping frames with previously predicted depth latents without added noise (consistent context-aware inference), plus a sequential spatial-then-temporal training protocol.
Key results
- On KITTI-360, ChronoDepth reports MFC 0.407, versus 0.655 for DepthCrafter and 0.952 for Depth Anything, a roughly 98% relative improvement in temporal consistency claimed by the authors.
- ChronoDepth achieves the best average rank across the three datasets (1.00 by MFC-based ranking, reported as overall best), with AbsRel 0.215 on KITTI-360 comparable to Depth Anything (0.215).
- Consistent context-aware inference improves ScanNet++ AbsRel from 0.194 (naive sliding window) and 0.194 (replacement trick) to 0.176 with MFC 0.092.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add single-frame depth datasets to multi-frame training (ablation) | ChronoDepth · ScanNet++ | AbsRel: 0.201 -> 0.176 | Helped |
| sample random rather than fixed clip lengths during training (ablation) | ChronoDepth · KITTI-360 | MFC: 0.470 -> 0.407 | Helped |
| train spatial and temporal layers sequentially, freezing spatial layers for temporal training (ablation) | ChronoDepth · ScanNet++ | AbsRel: 0.224 -> 0.176 | Helped |
| use consistent context-aware rather than naive sliding-window inference (ablation) | ChronoDepth · KITTI-360 | MFC: 0.505 -> 0.407 | Helped |
| use consistent context-aware rather than replacement-trick inference (ablation) | ChronoDepth · KITTI-360 | MFC: 0.479 -> 0.407 | Helped |
| use consistent context-aware inference (ablation) | ChronoDepth · ScanNet++ | MFC: 0.098 -> 0.092 | Helped |
Surprising findings
- The replacement trick used by DepthCrafter barely improves temporal consistency over naive sliding window inference in the ablation.
Limitations
- Relies on single-image datasets and a small set of synthetic video datasets, limiting scene diversity.
- Spatial accuracy on Sintel and KITTI-360 trails some baselines in AbsRel, so the gain is mainly in temporal consistency.
Experimental setup
Datasets: Hypersim TartanAir Virtual KITTI 2 MVS-Synth KITTI-360 ScanNet++ Sintel
Benchmarks: KITTI-360 ScanNet++ Sintel
Metrics: AbsRel delta1 accuracy multi-frame consistency (MFC)
Baselines: Marigold Marigold (SVD) Depth Anything Depth Anything V2 NVDS DepthCrafter
Builds on: Stable Video Diffusion Marigold EDM Diffusion Forcing
Closest papers by meaning
- Video Depth Anything: Consistent Depth Estimation for Super-Long Videos (CVPR 2025 · similarity 0.89)
- Depth Any Video with Scalable Synthetic Data (ICLR 2025 · similarity 0.89)
- Buffer Anytime: Zero-Shot Video Depth and Normal from Image Priors (CVPR 2025 · similarity 0.88)
- One-Step Diffusion for Detail-Rich and Temporally Consistent Video Super-Resolution (NeurIPS 2025 · similarity 0.87)
- GeoVideo: Introducing Geometric Regularization into Video Generation Model (NeurIPS 2025 · similarity 0.86)
- Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.