Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
FoundationStereo: Zero-Shot Stereo Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds FoundationStereo, a stereo depth model trained on a 1M-pair synthetic dataset with self-curation, using a side-tuning adapter over DepthAnythingV2 and attentive hybrid cost filtering to achieve zero-shot stereo generalization.
Problem
Stereo matching models typically require per-domain fine-tuning and generalize poorly zero-shot; the paper asks whether foundation-model-style zero-shot generalization is achievable for stereo depth estimation.
Approach
Combine a large, diverse, photorealistic synthetic stereo dataset with iterative self-curation, a side-tuning adapter that injects frozen DepthAnythingV2 monocular priors into the stereo feature extractor, and an attentive hybrid cost filter (axial-planar 3D convolutions plus disparity-wise transformer) feeding GRU iterative refinement.
Key results
- Zero-shot on Middlebury BP-2 1.1 vs. best prior second-block 7.5 (Selective-IGEV*) and ETH3D BP-1 0.5 vs 3.3 for best prior (Former-RAFT-DAM), trained with allowed datasets.
- Trained only on Scene Flow, zero-shot results already beat prior methods on all four benchmarks (e.g., Middlebury BP-2 5.5 vs 8.8 for IGEV).
- Scene Flow in-domain EPE 0.34 vs 0.41 previous best (MoCha-Stereo); fine-tuned version ranks 1st on ETH3D leaderboard, zero-shot ETH3D BP-1 1.52 comparable to fine-tuned leaders.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FoundationStereo instead of the strongest listed baseline | Zero-shot stereo matching · Middlebury | BP-2: 7.5 -> 1.1 | Helped |
| Use FoundationStereo instead of the strongest listed baseline | Zero-shot stereo matching · ETH3D | BP-1: 3.0 -> 0.5 | Helped |
| Use FoundationStereo instead of the strongest listed baseline | Zero-shot stereo matching · KITTI-12 | D1: 3.2 -> 2.3 | Helped |
| Use FoundationStereo instead of the strongest listed baseline | Zero-shot stereo matching · KITTI-15 | D1: 4.5 -> 2.8 | Helped |
| Use FoundationStereo instead of MoCha-Stereo | Scene Flow stereo matching · Scene Flow test | EPE: 0.41 -> 0.34 | Helped |
| Fine-tune on ETH3D training data | FoundationStereo · ETH3D test | EPE: 0.13 -> 0.09 | Helped |
| Use DepthAnythingV2-L instead of DINOv2-L (ablation) | STA · Middlebury training set | BP-2: 2.46 -> 1.97 | Helped |
| Use STA design (c) instead of STA design (a) (ablation) | FoundationStereo · Middlebury training set | BP-2: 6.48 -> 1.97 | Helped |
| Freeze the ViT instead of unfreezing it (ablation) | STA · Middlebury training set | BP-2: 3.94 -> 1.97 | Helped |
| Use cosine positional embedding instead of RoPE (ablation) | Disparity Transformer · Middlebury training set | BP-2: 2.19 -> 1.97 | Helped |
| Apply attention only along disparity rather than over the full cost volume (ablation) | Disparity Transformer · Middlebury training set | BP-2: 2.25 -> 1.97 | Helped |
| Add STA (ablation) | FoundationStereo · Middlebury training set | BP-2: 2.34 -> 1.15 | Helped |
Surprising findings
- DINOv2-L was less effective than DepthAnythingV2 for stereo matching, and unfreezing the ViT degraded the pretrained monocular priors.
- Attention over the full 4D cost volume was less effective than attention restricted to the disparity dimension.
Limitations
- Not optimized for efficiency; 0.7s inference on 375x1242 on A100.
- Limited collection of transparent objects in the training dataset.
- Fine-tuned ETH3D results rely on in-domain data; zero-shot numbers on some benchmarks still depend on the mixture of training datasets.
Experimental setup
Datasets: FoundationStereo Dataset (FSD) Scene Flow Sintel CREStereo FallingThings InStereo2K Virtual KITTI 2
Benchmarks: Middlebury ETH3D KITTI-12 KITTI-15 Scene Flow ETH3D leaderboard
Metrics: BP-2 BP-1 D1 EPE BP-0.5
Baselines: CREStereo Selective-IGEV IGEV-Stereo NMRF CroCo-Stereo MoCha-Stereo RAFT-Stereo Former-RAFT-DAM
Builds on: RAFT-Stereo IGEV-Stereo DepthAnythingV2 ViT-Adapter DUSt3R
Closest papers by meaning
- Stereo Anywhere: Robust Zero-Shot Deep Stereo Matching Even Where Either Stereo or Mono Fail (CVPR 2025 · similarity 0.89)
- DEFOM-Stereo: Depth Foundation Model Based Stereo Matching (CVPR 2025 · similarity 0.88)
- Any-Stereo: Arbitrary Scale Disparity Estimation for Iterative Stereo Matching (AAAI 2024 · similarity 0.87)
- BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation (NeurIPS 2024 · similarity 0.87)
- Stereo Risk: A Continuous Modeling Approach to Stereo Matching (ICML 2024 · similarity 0.87)
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.