Research map / Label-efficient and robust learning / Recognition and long-tail
Adaptive Multi-Modal Cross-Entropy Loss for Stereo Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an adaptive multi-modal cross-entropy loss that models stereo ground truth as a mixture of Laplacians from DBSCAN clusters in a local window, plus a dominant-modal disparity estimator, improving PSMNet, GwcNet and GANet on SceneFlow, KITTI and cross-domain tests.
Problem
Stereo matching networks trained with cross-entropy loss assume uni-modal ground-truth distributions per pixel, which fails at multi-modal edge pixels and causes over-smoothing, bleeding and misalignment artifacts.
Approach
Cluster disparities in a local window with DBSCAN and model each pixel's ground-truth as a weighted mixture of Laplacians (one per cluster), so edge pixels get multi-modal targets; pair this with a dominant-modal disparity estimator that picks the modal with largest cumulative probability.
Key results
- GANet + Ours ranks 1st on KITTI 2015 and KITTI 2012 among published methods (D1-all 1.38 on KITTI 2015 All, 1.24 on KITTI 2012 Noc Out).
- On SceneFlow, PSMNet + Ours reduces EPE from 1.09 to 0.64 (41.28% improvement).
- Trained only on SceneFlow, GANet + Ours reaches 2.31% on ETH3D (>1px), beating prior generalization methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace smooth L1 with adaptive multi-modal cross-entropy and use DME | PSMNet · SceneFlow test | EPE: 0.97 -> 0.78 | Helped |
| Replace uni-modal cross-entropy with multi-modal cross-entropy (ablation) | PSMNet with SME · SceneFlow test | EPE: 0.84 -> 0.80 | Helped |
| Replace SME with DME (ablation) | PSMNet with uni-modal cross-entropy · SceneFlow test | EPE: 0.84 -> 0.82 | Helped |
| Replace SME with DME (ablation) | PSMNet with multi-modal cross-entropy · SceneFlow test | EPE: 0.80 -> 0.78 | Helped |
| Use a 3 × 3 rather than 1 × 9 local window (ablation) | PSMNet + Ours · SceneFlow test | EPE: 0.78 -> 0.83 | Hurt |
| Use a 9 × 1 rather than 1 × 9 local window (ablation) | PSMNet + Ours · SceneFlow test | EPE: 0.78 -> 0.84 | Hurt |
| Vary the 1D horizontal window size from 1 × 9 to 1 × 3, 1 × 5, 1 × 7, or 1 × 11 (ablation) | PSMNet + Ours · SceneFlow test | EPE: 1 × 3: 0.82; 1 × 5: 0.81; 1 × 7: 0.81; 1 × 9: 0.78; 1 × 11: 0.84 | Mixed |
| Reduce training ground-truth density to 20% (ablation) | PSMNet · KITTI 2015 validation | D1: 1.90 -> 2.35 | Hurt |
| Reduce training ground-truth density to 20% (ablation) | PSMNet + Ours · KITTI 2015 validation | D1: 1.58 -> 1.74 | Hurt |
| Replace baseline networks' training loss and disparity estimator with the proposed method | PSMNet, GwcNet, and GANet · SceneFlow test | EPE: 41.28%, 19.48%, and 35.90% | Helped |
| Apply the proposed method to GANet | GANet · KITTI 2015 and KITTI 2012 | D1-all (KITTI 2015); Out-All (KITTI 2012): 1.63 -> 1.40; 1.60 -> 1.29 | Helped |
| Train with the proposed loss and estimator instead of the baseline method | GANet · ETH3D cross-domain | >1px: 14.1 -> 2.31 | Helped |
Surprising findings
- Even with only 20% of the ground-truth density, PSMNet + Ours outperforms PSMNet trained on 100% density on KITTI 2015.
Limitations
- Relies on hand-tuned window shape/size and DBSCAN thresholds (eps=3, minPts=1), and a larger 3x9 window is needed for sparse KITTI ground truth.
- Multi-modal outputs at edges still cause some ambiguity; gains on KITTI are modest for PSMNet relative to iterative methods like IGEVStereo.
Experimental setup
Datasets: SceneFlow KITTI 2012 KITTI 2015
Benchmarks: SceneFlow KITTI 2015 KITTI 2012 Middlebury ETH3D
Metrics: EPE 1px error 3px error D1 outlier rate
Baselines: PSMNet GwcNet GANet PSMNet + SME (uni-modal cross-entropy) AcfNet ACVNet IGEVStereo LEAStereo CroCoStereo DSMNet CFNet ITSA-CFNet
Builds on: PSMNet GwcNet GANet SME (single-modal disparity estimator) DBSCAN
Closest papers by meaning
- Active-Passive SimStereo - Benchmarking the Cross-Generalization Capabilities of Deep Learning-based Stereo Methods (NeurIPS 2022 · similarity 0.87)
- Learning the Distribution of Errors in Stereo Matching for Joint Disparity and Uncertainty Estimation (CVPR 2023 · similarity 0.87)
- Modeling Stereo-Confidence out of the End-to-End Stereo-Matching Network via Disparity Plane Sweep (AAAI 2024 · similarity 0.87)
- Improved Stereo Matching With Constant Highway Networks and Reflective Confidence Learning (CVPR 2017 · similarity 0.87)
- Stereo Risk: A Continuous Modeling Approach to Stereo Matching (ICML 2024 · similarity 0.86)
- Robust Synthetic-to-Real Transfer for Stereo Matching (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.