Research map / 3D vision / Depth and stereo
DEFOM-Stereo: Depth Foundation Model Based Stereo Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Integrates Depth Anything V2 (as a depth foundation model) into RAFT-Stereo via combined CNN+DPT feature/context encoders and a scale update module that refines monocular-depth-initialized disparity, achieving strong zero-shot and benchmark stereo results.
Problem
Stereo matching struggles with occlusion, textureless regions, blur and high resolution, and generalizes poorly zero-shot; monocular relative depth models generalize well but output affine-invariant depth with inconsistent scale, so they cannot directly supply metric disparity.
Approach
Fuse features from a pretrained depth foundation model (Depth Anything V2 DPT/ViT features) with CNN encoders for matching and context, initialize recurrent disparity from the monocular depth normalized by image width, and add a scale update module with a scale lookup over the full correlation volume before the usual delta update.
Key results
- Zero-shot (Scene Flow only): DEFOM-Stereo ViT-L reduces error vs RAFT-Stereo by about 13% on KITTI, 28% on ETH3D, and over half on Middlebury (half resolution).
- On the KITTI 2012 leaderboard it ranks 1st on several metrics; on Middlebury it ranks first in 7 of 10 all-pixel metrics, outperforming Selective-IGEV by 16.9% on Bad 2.0 (all).
- On ETH3D, reduces Bad 1.0 (noc) by 23.1% and Bad 1.0 (all) by 24.3% relative to LoS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Integrate Depth Anything V2 features and use its predicted depth to initialize recurrent disparity, with scale updates | RAFT-Stereo; DEFOM-Stereo (ViT-L) · KITTI 2012 train, zero-shot | 3-pixel error rate: 4.35 -> 3.76 | Helped |
| Integrate Depth Anything V2 features and use its predicted depth to initialize recurrent disparity, with scale updates | RAFT-Stereo; DEFOM-Stereo (ViT-L) · KITTI 2015 train, zero-shot | 3-pixel error rate: 5.74 -> 4.99 | Helped |
| Integrate Depth Anything V2 features and use its predicted depth to initialize recurrent disparity, with scale updates | RAFT-Stereo; DEFOM-Stereo (ViT-L) · Middlebury full, zero-shot | 2-pixel error rate: 18.33 -> 11.95 | Helped |
| Integrate Depth Anything V2 features and use its predicted depth to initialize recurrent disparity, with scale updates | RAFT-Stereo; DEFOM-Stereo (ViT-L) · ETH3D train, zero-shot | 1-pixel error rate: 3.28 -> 2.35 | Helped |
| Use DEFOM-Stereo instead of StereoBase | KITTI 2012 benchmark · KITTI 2012 | Bad 2.0, all (%): 1.95 -> 1.79 | Helped |
| Use DEFOM-Stereo instead of StereoBase | KITTI 2015 benchmark · KITTI 2015 | D1-all (%): 1.44 -> 1.411 | Helped |
| Use DEFOM-Stereo instead of Selective-IGEV | Middlebury benchmark · Middlebury | Bad 2.0, all (%): 6.04 -> 5.021 | Helped |
| Use DEFOM-Stereo instead of LoS | ETH3D benchmark · ETH3D | Bad 1.0, all (%): 1.03 -> 0.782 | Helped |
| Use DEFOM-Stereo RVC instead of CREStereo++ RVC | Robust Vision Challenge models · Middlebury, RVC | Bad 2.0, all (%): 9.46 -> 6.90 | Helped |
Surprising findings
- Later recurrent methods (DLNR, IGEV-based models) improve Scene Flow fitting but fail to improve zero-shot generalization comprehensively, with some increasing error on KITTI and ETH3D.
- Depth Anything V2 affine-aligned disparity shows large scale inconsistency within images, especially on synthetic Scene Flow data.
Limitations
- Scale inconsistency of monocular depth on synthetic stereo data requires an extra scale update module.
- Benchmark results rely on dataset-specific multi-stage finetuning mixtures.
Experimental setup
Datasets: Scene Flow KITTI 2012 KITTI 2015 Middlebury ETH3D Tartan Air CREStereo Dataset Falling Things InStereo2k CARLA HR-VS virtual KITTI 2 Sintel Stereo
Benchmarks: KITTI 2012 KITTI 2015 Middlebury ETH3D Robust Vision Challenge
Metrics: Bad 2.0 error rate Bad 3.0 error rate D1-all AvgErr RMS Bad 1.0 error rate
Baselines: RAFT-Stereo IGEV-Stereo Selective-IGEV Mocha-Stereo NMRF-Stereo DLNR DSMNet ViTAStereo StereoBase LoS CREStereo CREStereo++
Builds on: RAFT-Stereo Depth Anything V2 DPT DINOv2 IGEV-Stereo
Closest papers by meaning
- FoundationStereo: Zero-Shot Stereo Matching (CVPR 2025 · similarity 0.88)
- Stereo Anywhere: Robust Zero-Shot Deep Stereo Matching Even Where Either Stereo or Mono Fail (CVPR 2025 · similarity 0.88)
- Any-Stereo: Arbitrary Scale Disparity Estimation for Iterative Stereo Matching (AAAI 2024 · similarity 0.87)
- Stereo Risk: A Continuous Modeling Approach to Stereo Matching (ICML 2024 · similarity 0.87)
- MonSter: Marry Monodepth to Stereo Unleashes Power (CVPR 2025 · similarity 0.86)
- Rectification-specific Supervision and Constrained Estimator for Online Stereo Rectification (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.