Research map / 3D vision / Depth and stereo
MSP-MVS: Multi-Granularity Segmentation Prior Guided Multi-View Stereo
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MSP-MVS, a PatchMatch-based multi-view stereo method that uses multi-granularity segmentation edges from Semantic-SAM as priors to guide edge-confined deformable patches, with attention-consistent anchor clustering and 3D disparity-sampling optimization.
Problem
Deformable patch-based multi-view stereo can skip depth edges and cross into heterogeneous areas, causing matching distortions and imbalanced anchor attention, and fixed sampling can trap matching costs in local minima.
Approach
Aggregate and CRF-refine boundaries from Semantic-SAM masks at scene, instance and feature granularities to constrain anchor selection to homogeneous areas; add adaptive equidistribution of sectors, disassemble-clustering of anchors, and iterative 3D optimization over sampling pixels and disparities.
Key results
- On ETH3D at 2cm, MSP-MVS reaches F1 88.09 on training and 89.51 on test, the highest F1 among compared methods (APD-MVS base 87.44 on test).
- On TNT intermediate, MSP-MVS reaches F1 64.48, best among compared methods (EPNet 63.68, APD-MVS 63.64).
- Runtime is about one-third higher than the APD-MVS baseline, while GPU memory is lower than ACMMP, SD-MVS and HPM-MVS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add edge aggregation and edge refinement (ablation) | MSP-MVS · ETH3D 2cm | F1: 86.53 -> 88.09 | Helped |
| Add edge aggregation (ablation) | MSP-MVS · ETH3D 2cm | F1: 87.07 -> 88.09 | Helped |
| Add edge refinement (ablation) | MSP-MVS · ETH3D 2cm | F1: 87.54 -> 88.09 | Helped |
| Add adaptive equidistribution and disassemble-clustering (ablation) | MSP-MVS · ETH3D 2cm | F1: 86.61 -> 88.09 | Helped |
| Add adaptive equidistribution (ablation) | MSP-MVS · ETH3D 2cm | F1: 87.23 -> 88.09 | Helped |
| Add disassemble-clustering (ablation) | MSP-MVS · ETH3D 2cm | F1: 87.29 -> 88.09 | Helped |
| Add disparity-sampling synergistic 3D optimization (ablation) | MSP-MVS · ETH3D 2cm | F1: 86.94 -> 88.09 | Helped |
| Add the variance component to the optimization objective (ablation) | MSP-MVS · ETH3D 2cm | F1: 87.17 -> 88.09 | Helped |
| Add the cost component to the optimization objective (ablation) | MSP-MVS · ETH3D 2cm | F1: 87.49 -> 88.09 | Helped |
| Use μ = 5 rather than μ = 3 (ablation) | MSP-MVS · ETH3D 2cm | F1: 87.73 -> 88.09 | Helped |
| Use adaptive sampling rather than fixed-interval sampling (ablation) | MSP-MVS · ETH3D 2cm | F1: 86.94 -> 88.09 | Helped |
| Use global sampling rather than adaptive sampling (ablation) | MSP-MVS · ETH3D 2cm | F1: 88.09 -> 88.45 | Mixed |
Surprising findings
- On TNT advanced, MSP-MVS falls short of EPNet by 0.26% in F1, placing second.
- Global sampling achieves slightly higher F1 than the adaptive sampling used by the method, but needs over 5 times the runtime.
Limitations
- Runtime is higher than the APD-MVS baseline.
- Slightly behind EPNet on TNT advanced F1.
- Relies on the segmentation quality of Semantic-SAM for depth edges.
Experimental setup
Datasets: ETH3D Tanks and Temples
Benchmarks: ETH3D Tanks and Temples
Metrics: F1 completeness accuracy precision recall runtime GPU memory
Baselines: PatchMatchNet IterMVS-LS MVSTER EPP-MVSNet EPNet AA-RMVSNet TAPA-MVS PCF-MVS ACMM ACMP ACMMP SD-MVS
Builds on: APD-MVS PatchMatch Semantic-SAM ACMMP DBSCAN
Closest papers by meaning
- DVP-MVS: Synergize Depth-Edge and Visibility Prior for Multi-View Stereo (AAAI 2025 · similarity 0.91)
- Dual-Level Precision Edges Guided Multi-View Stereo with Accurate Planarization (AAAI 2025 · similarity 0.90)
- SD-MVS: Segmentation-Driven Deformation Multi-View Stereo with Spherical Refinement and EM Optimization (AAAI 2024 · similarity 0.90)
- Adaptive Patch Deformation for Textureless-Resilient Multi-View Stereo (CVPR 2023 · similarity 0.89)
- Learning Deformable Hypothesis Sampling for Accurate PatchMatch Multi-View Stereo (AAAI 2024 · similarity 0.85)
- PatchFusion: An End-to-End Tile-Based Framework for High-Resolution Monocular Metric Depth Estimation (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.