Research map / Video / Video segmentation and tracking
Buffer Anytime: Zero-Shot Video Depth and Normal from Image Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Buffer Anytime fine-tunes single-image depth (Depth Anything V2) and normal (Marigold-E2E-FT) models into temporally consistent video estimators using only image priors plus optical-flow smoothness loss, with no paired video-geometry data.
Problem
Video depth and surface normal estimation suffers from frame-to-frame inconsistency in image models, while existing temporally consistent video models require large paired video-geometry annotated datasets.
Approach
Zero-shot training that keeps a frozen single-frame image model as a regularizer on one randomly sampled frame, adds an optical-flow-warped temporal stabilization loss with cycle-consistency and depth-edge masking, and uses a lightweight zero-initialized temporal attention module inserted into the image backbone.
Key results
- On video depth, Buffer Anytime (Depth Anything V2 backbone) reaches AbsRel 0.123 on ScanNet and 0.119 on KITTI, improving over the backbone's 0.135 and 0.140 and approaching DepthCrafter (0.125, 0.110).
- On KITTI, OPW temporal error drops from 0.089 (Depth Anything V2) to 0.038 for the proposed model.
- On video normal estimation, OPW on Sintel falls from 0.152 (Marigold-E2E-FT) to 0.065 while mean angular error stays comparable (33.421 vs 32.858).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add temporal attention and optical-flow stabilization to Depth Anything V2 | Depth Anything V2 · ScanNet | OPW: 0.121 -> 0.076 | Helped |
| Add temporal attention and optical-flow stabilization to Depth Anything V2 | Depth Anything V2 · KITTI | AbsRel: 0.140 -> 0.119 | Helped |
| Add temporal attention and optical-flow stabilization to Depth Anything V2 | Depth Anything V2 · KITTI | OPW: 0.089 -> 0.038 | Helped |
| Add temporal attention and optical-flow stabilization to Depth Anything V2 | Depth Anything V2 · Bonn | OPW: 0.059 -> 0.028 | Helped |
| Add temporal attention and optical-flow stabilization to Marigold-E2E-FT | Marigold-E2E-FT · Sintel | OPW: 0.152 -> 0.065 | Helped |
| Add temporal attention and optical-flow stabilization to Marigold-E2E-FT | Marigold-E2E-FT · ScanNet | OPW: 0.092 -> 0.069 | Helped |
| Add temporal attention and optical-flow stabilization to Marigold-E2E-FT | Marigold-E2E-FT · Sintel | Mean angular error: 32.858 -> 33.421 | Hurt |
| Set regularization weight ωreg. = 0.1 (ablation) | Buffer Anytime depth model · KITTI | AbsRel: 0.119 -> 0.137 | Hurt |
| Set regularization weight ωreg. = 3 (ablation) | Buffer Anytime depth model · KITTI | AbsRel: 0.119 -> 0.122 | Hurt |
| Remove optical flow correspondence masking (ablation) | Buffer Anytime depth model · KITTI | AbsRel: 0.119 -> 0.122 | Hurt |
| Apply regularization to all frames instead of one sampled frame (ablation) | Buffer Anytime depth model · KITTI | AbsRel: 0.119 -> 0.120 | No clear effect |
Surprising findings
- Using all frames for regularization instead of a single random frame gave similar performance to the standard model, suggesting single-frame regularization suffices.
- The depth model improves AbsRel over its image backbone while also improving temporal smoothness, rather than trading one for the other.
Limitations
- Inherits failures of the backbone image model in extreme cases.
- Optical flow only captures correlations between adjacent frames, so objects leaving and re-entering the scene may be inconsistent.
- Normal evaluation lacks a standard video benchmark; protocol relies on image metrics plus OPW.
Experimental setup
Datasets: in-house unlabeled video clips (~200K) ScanNet KITTI Bonn Sintel
Benchmarks: ScanNet KITTI Bonn Sintel
Metrics: AbsRel delta1 OPW mean angular error median angular error 11.25 degree accuracy
Baselines: Depth Anything V2 ChronoDepth NVDS DepthCrafter Marigold Marigold-E2E-FT DSINE Lotus
Builds on: Depth Anything V2 Marigold-E2E-FT AnimateDiff GMFlow
Closest papers by meaning
- Video Depth Anything: Consistent Depth Estimation for Super-Long Videos (CVPR 2025 · similarity 0.91)
- Learning Temporally Consistent Video Depth from Video Diffusion Priors (CVPR 2025 · similarity 0.88)
- Taming generative video models for zero-shot optical flow extraction (NeurIPS 2025 · similarity 0.87)
- Depth Any Video with Scalable Synthetic Data (ICLR 2025 · similarity 0.86)
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (CVPR 2024 · similarity 0.86)
- Zero-Shot Monocular Scene Flow Estimation in the Wild (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.