Research map / 3D vision / Depth and stereo
SD-MVS: Segmentation-Driven Deformation Multi-View Stereo with Spherical Refinement and EM Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SD-MVS, a PatchMatch-based multi-view stereo method that uses SAM instance masks for adaptive patch deformation, spherical gradient refinement of normals and pixelwise depth search, and EM-based tuning of cost weights, reaching top ETH3D F1.
Problem
Multi-view stereo reconstruction struggles in textureless areas and with depth discontinuities, and PatchMatch-based methods depend on empirically tuned cost hyperparameters.
Approach
Use SAM instance segmentation to deform matching patches and propagation search domains so pixels only use neighbors within their own instance, refine normals on a sphere via two-axis rotations with gradient-guided directions, and alternate EM updates between aggregated cost and cost-term weights.
Key results
- On ETH3D test at 2cm threshold, SD-MVS reaches F1 88.06, the best among compared methods (APD-MVS 87.44).
- On Tanks and Temples intermediate set, SD-MVS F1 is 63.31, competitive with EPNet (63.68) and APD-MVS (63.64).
- Runtime is about half that of APD-MVS and ACMMP, though GPU memory is roughly one-third higher than those methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add matching-cost patch deformation and multi-scale consistency (ablation) | SD-MVS · ETH3D | F1: 81.27 -> 86.94 | Helped |
| add patch deformation to matching cost (ablation) | SD-MVS · ETH3D | F1: 83.42 -> 86.94 | Helped |
| add multi-scale matching-cost consistency (ablation) | SD-MVS · ETH3D | F1: 84.55 -> 86.94 | Helped |
| use the proposed propagation instead of ACMMP propagation (ablation) | SD-MVS · ETH3D | F1: 84.52 -> 86.94 | Helped |
| add patch deformation to propagation (ablation) | SD-MVS · ETH3D | F1: 85.38 -> 86.94 | Helped |
| add multi-scale consistency to propagation (ablation) | SD-MVS · ETH3D | F1: 85.54 -> 86.94 | Helped |
| add refinement (ablation) | SD-MVS · ETH3D | F1: 77.6 -> 86.94 | Helped |
| use spherical gradient refinement instead of Gipuma refinement (ablation) | SD-MVS · ETH3D | F1: 83.43 -> 86.94 | Helped |
| use spherical gradient refinement instead of ACMMP refinement (ablation) | SD-MVS · ETH3D | F1: 84.25 -> 86.94 | Helped |
| add EM-based optimization instead of fixed weights (1,0.5,0.5) (ablation) | SD-MVS · ETH3D | F1: 83.45 -> 86.94 | Helped |
| add EM-based optimization instead of fixed weights (1,0.2,0.2) (ablation) | SD-MVS · ETH3D | F1: 84.27 -> 86.94 | Helped |
Surprising findings
- Memory use is about one-third higher than APD-MVS and ACMMP despite the runtime gain.
Limitations
- Memory consumption higher than traditional SOTA methods
- Highlight areas and view selection not addressed (noted as future work)
- Tanks and Temples results trail EPNet and APD-MVS in F1
Experimental setup
Datasets: ETH3D high-resolution MVS benchmark Tanks and Temples
Benchmarks: ETH3D high-resolution multi-view stereo benchmark Tanks and Temples
Metrics: accuracy completeness F1 score GPU memory runtime
Baselines: PatchMatchNet IterMVS-LS MVSTER EPP-MVSNet EPNet COLMAP PCF-MVS MAR-MVS ACMP ACMMP APD-MVS
Builds on: ACMP ACMMP APD-MVS Segment Anything Model Gipuma
Closest papers by meaning
- MSP-MVS: Multi-Granularity Segmentation Prior Guided Multi-View Stereo (AAAI 2025 · similarity 0.90)
- DVP-MVS: Synergize Depth-Edge and Visibility Prior for Multi-View Stereo (AAAI 2025 · similarity 0.89)
- Adaptive Patch Deformation for Textureless-Resilient Multi-View Stereo (CVPR 2023 · similarity 0.88)
- Dual-Level Precision Edges Guided Multi-View Stereo with Accurate Planarization (AAAI 2025 · similarity 0.88)
- Multi-View Stereo Representation Revist: Region-Aware MVSNet (CVPR 2023 · similarity 0.87)
- RRT-MVS: Recurrent Regularization Transformer for Multi-View Stereo (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.