Research map / Video / Video segmentation and tracking
CH3Depth: Efficient and Flexible Depth Foundation Model with Flow Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CH3Depth, a flow-matching depth foundation model that uses InDI-style flow matching, non-uniform sampling, and a Latent Temporal Stabilizer to deliver accurate, efficient image and temporally consistent video depth estimation.
Problem
Existing depth foundation models struggle to jointly deliver fine detail, temporal consistency for video, and high inference efficiency; prior flow-matching depth models (DepthFM) use globally fixed velocity fields and uniform sampling steps.
Approach
Replaces the fixed-scale flow matching velocity target with InDI (predicting the transport from current state to target, equivalent to reweighting by 1-t), uses a concave non-uniform timestep schedule to reduce steps, and adds a lightweight latent-space sliding-window UNet (LTS) trained with a temporal consistent deviation loss on natural videos.
Key results
- Image depth: CH3Depth (96K training) reaches 5.0 AbsRel / 97.2 delta1 on NYUv2 and 7.6 AbsRel on KITTI, versus Marigold 5.5/96.4 and 9.9 on KITTI.
- Video depth on Sintel: CH3Depth with LTS gets 19.4 AbsRel and 73.5 delta1 with OPW 40.7, versus DepthCrafter 29.2 AbsRel and 33.6 OPW; on NYUv2 OPW 9.5 vs DepthCrafter 8.8.
- LTS reduces OPW error by 60.3% on the flow matching baseline (Table 4), and transferring LTS to Marigold and DepthFM lowers Sintel OPW substantially (e.g., DepthFM 87.9 to 50.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace FM-OT with InDI flow matching (ablation) | CH3Depth · NYUv2 | AbsRel: 6.4 -> 5.2 | Helped |
| add non-uniform sampling (ablation) | CH3Depth with InDI · NYUv2 | AbsRel: 5.2 -> 5.1 | Helped |
| add LTS (ablation) | CH3Depth with InDI and non-uniform sampling · NYUv2 | OPW: 30.7 -> 12.2 | Helped |
| add LTS (ablation) | CH3Depth with InDI and non-uniform sampling · NYUv2 | AbsRel: 5.1 -> 5.4 | Hurt |
| add consistent deviation loss (ablation) | CH3Depth with LTS · NYUv2 | OPW: 12.2 -> 9.5 | Helped |
| use concave non-uniform sampling f(x)=x^(1/2) instead of uniform sampling (ablation) | CH3Depth · NYUv2 | AbsRel: 5.3 -> 5.1 | Helped |
| vary the LTS window size from w=3 to w=5 (ablation) | CH3Depth · NYUv2 | OPW: 9.9; 9.5; 9.4 | Mixed |
| add LTS (ablation) | Marigold · NYUv2 | OPW: 45.5 -> 27.9 | Helped |
| add LTS (ablation) | DepthFM · NYUv2 | OPW: 38.4 -> 19.1 | Helped |
| use 96K rather than 74K training samples | CH3Depth · NYUv2 | AbsRel: 5.2 -> 5.0 | Helped |
| use 96K rather than 74K training samples | CH3Depth · KITTI | AbsRel: 8.0 -> 7.6 | Helped |
Surprising findings
- Plain image-level flow matching with LTS-free inference gives strong accuracy, but temporal consistency error (OPW) on Sintel for CH3Depth without LTS is high (30.7 on NYUv2 setting), while LTS cuts it dramatically.
Limitations
- Fails in challenging scenes such as mirror reflections and extreme lighting, as acknowledged by the authors.
- Video consistency on Sintel is still weaker than DepthCrafter in OPW (40.7 vs 33.6), and LTS adds parameters (1560.7M total).
Experimental setup
Datasets: Hypersim Virtual KITTI 2 TartanAir SHIFT MVS-Synth VDW WSVD Sintel NYUv2 KITTI ETH3D ScanNet
Benchmarks: NYUv2 KITTI ETH3D ScanNet Sintel
Metrics: AbsRel delta1 accuracy OPW temporal consistency error runtime per step parameter count
Baselines: Marigold DepthFM Lotus Depth Anything GeoWizard DepthCrafter ChronoDepth NVDS
Builds on: DepthFM Marigold InDI Stable Diffusion VAE NVDS
Closest papers by meaning
- DepthFM: Fast Generative Monocular Depth Estimation with Flow Matching (AAAI 2025 · similarity 0.88)
- Video Depth Anything: Consistent Depth Estimation for Super-Long Videos (CVPR 2025 · similarity 0.87)
- BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation (NeurIPS 2024 · similarity 0.86)
- FoundationStereo: Zero-Shot Stereo Matching (CVPR 2025 · similarity 0.86)
- Learning Temporally Consistent Video Depth from Video Diffusion Priors (CVPR 2025 · similarity 0.86)
- Align3R: Aligned Monocular Depth Estimation for Dynamic Videos (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.