Research map / Probabilistic and scientific modeling / Molecules and proteins

Neural Markov Random Field for Stereo Matching

CVPR 2024 · Probabilistic and scientific modeling › Molecules and proteins · approach: Neuro-symbolic probabilistic layers · 44 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a Neural MRF (NMRF) for stereo matching in which pairwise potentials and message passing are learned via neural networks guided by mean-field variational inference, plus a Disparity Proposal Network that prunes the disparity search space.

Problem

Stereo matching with hand-crafted MRF pairwise potentials and message passing limits accuracy compared with end-to-end deep models; MRF inference is also intractable at high resolution without pruning the disparity label space.

Approach

Fully data-driven MRF in which each disparity label is an embedding; unary and pairwise potentials and mean-field message passing are replaced by neural modules (transformer-style attention with relative positional bias, separate self-edge and neighbor-edge potentials), with a DPN that proposes multi-modal candidate disparities per pixel.

Key results

• Ranks 1st on KITTI 2012 and 2015 leaderboards among published methods while running under 100 ms (0.09 s on KITTI 2015 per Table 2).
• KITTI 2015 D1-bg 1.28% vs 2.85% for LBPS, a reduction of more than 50%.
• Zero-shot from SceneFlow: KITTI-12 4.2, KITTI-15 5.1, Middlebury 7.5, ETH3D 3.8, surpassing DSMNet and CREStereo++ on the listed datasets.

Measured effects reported in the paper

Surprising findings

• Using only one candidate label per pixel still gives competitive performance, and the method is insensitive to the number of candidate labels.
• Message passing along self edges (intra-pixel label competition), usually overlooked in hand-crafted MRFs, gives notable gains (EPE 0.57 to 0.53 on SceneFlow).

Limitations

• Overall accuracy gains depend on refinement stage; cascaded refinement trades off generalization for performance.
• Candidate label quality bounds performance; DPN occasionally fails in textureless or occluded regions.

Experimental setup

Datasets: SceneFlow KITTI 2012 KITTI 2015 Middlebury 2014 ETH3D

Benchmarks: SceneFlow KITTI 2012 KITTI 2015 Middlebury 2014 ETH3D

Metrics: EPE Bad 1.0 bad 2.0 bad 3.0 D1-all outlier ratio runtime

Baselines: LEAStereo IGEV-Stereo ACVNet GANet-deep PSMNet RAFT-Stereo LBPS PBCP CREStereo++ DSMNet

Builds on: Markov Random Field mean-field variational inference DETR Transformer self-attention CSWin cross-shaped window attention

Closest papers by meaning

• End-To-End Training of Hybrid CNN-CRF Models for Stereo (CVPR 2017 · similarity 0.87)
• Zoom and Learn: Generalizing Deep Stereo Matching to Novel Domains (CVPR 2018 · similarity 0.87)
• Hierarchical Neural Architecture Search for Deep Stereo Matching (NeurIPS 2020 · similarity 0.87)
• Stereo Risk: A Continuous Modeling Approach to Stereo Matching (ICML 2024 · similarity 0.87)
• All-in-One: Transferring Vision Foundation Models into Stereo Matching (AAAI 2025 · similarity 0.86)
• Modeling Stereo-Confidence out of the End-to-End Stereo-Matching Network via Disparity Plane Sweep (AAAI 2024 · similarity 0.86)