Research map / Label-efficient and robust learning / Recognition and long-tail
All-in-One: Transferring Vision Foundation Models into Stereo Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AIO-Stereo, which distills and selectively fuses knowledge from multiple heterogeneous vision foundation models (DINOv2, SAM, Depth Anything v2) into a CNN-based iterative stereo matching network.
Problem
Stereo matching feature extractors are weak in general representation; transferring knowledge from multiple heterogeneous (Transformer-based) vision foundation models into a CNN stereo model is hard due to architectural mismatch and conflicting knowledge.
Approach
A dual-level knowledge utilization (expert network plus feature alignment network for distillation, and residual feature addition for forward fusion) combined with a MoE-style top-k gating module that selects per-pixel knowledge from each VFM's expert.
Key results
- Ranks 1st on the Middlebury leaderboard with bad 2.0 of 2.36 (vs. 2.51 for Selective-IGEV) and bad 1.0 of 6.08.
- On ETH3D, achieves bad 1.0 of 0.94, outperforming Selective-IGEV (1.23) by 23.58% on bad 1.0.
- Ablation on MiddEval v3 training set: full model reaches EPE 0.66 and bad-2px 3.48% vs. baseline 0.74 and 4.68%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add selective expert selection (ablation) | AIO-Stereo · MiddEval v3 training set, full resolution | EPE: 0.68 -> 0.66 | Helped |
| Add selective expert selection (ablation) | AIO-Stereo · MiddEval v3 training set, full resolution | >2px (%): 3.57 -> 3.48 | Helped |
| Add aligned knowledge distillation (ablation) | AIO-Stereo · MiddEval v3 training set, full resolution | >2px (%): 3.87 -> 3.48 | Helped |
| Add forward fusion (ablation) | AIO-Stereo · MiddEval v3 training set, full resolution | EPE: 0.67 -> 0.66 | Helped |
| Add forward fusion (ablation) | AIO-Stereo · MiddEval v3 training set, full resolution | >2px (%): 3.52 -> 3.48 | Helped |
| Add SAM and Depth Anything v2 to DINO (ablation) | AIO-Stereo · MiddEval v3 training set, full resolution | >2px (%): 3.64 -> 3.48 | Helped |
| Add Depth Anything v2 to DINO and SAM (ablation) | AIO-Stereo · MiddEval v3 training set, full resolution | >2px (%): 3.61 -> 3.48 | Helped |
Surprising findings
- Naively merging or distilling VFM features into the stereo model hurts performance (reported -0.71 EPE in experiments) and causes unstable training.
- Non-selective transfer that accepts all VFM knowledge underperforms the selective gating variant.
Limitations
- Only zero-shot evaluation on Middlebury is reported; the paper gives no runtime or parameter cost analysis of the three VFMs at training and inference.
- Gains on KITTI 2015 are comparable rather than clearly best.
Experimental setup
Datasets: Scene Flow Middlebury 2014 KITTI-2015 KITTI-2012 ETH3D TartanAir CREStereo Dataset Falling Things InStereo2k CARLA HR-VS Sintel Stereo
Benchmarks: Middlebury ETH3D KITTI 2015 Scene Flow
Metrics: bad 2.0 bad 1.0 avgerr A90 EPE D1-all D1-bg D1-fg
Baselines: PSMNet HITNet RAFT-Stereo LEAStereo CREStereo GMStereo IGEV-Stereo DLNR Selective-IGEV EAI-Stereo
Builds on: Selective-Stereo IGEV-Stereo RAFT-Stereo OFA-KD
Closest papers by meaning
- Hierarchical Object-Aware Dual-Level Contrastive Learning for Domain Generalized Stereo Matching (NeurIPS 2024 · similarity 0.88)
- FoundationStereo: Zero-Shot Stereo Matching (CVPR 2025 · similarity 0.87)
- Stereo Risk: A Continuous Modeling Approach to Stereo Matching (ICML 2024 · similarity 0.86)
- Neural Markov Random Field for Stereo Matching (CVPR 2024 · similarity 0.86)
- Any-Stereo: Arbitrary Scale Disparity Estimation for Iterative Stereo Matching (AAAI 2024 · similarity 0.86)
- Learning Intra-view and Cross-view Geometric Knowledge for Stereo Matching (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.