Research map / 3D vision / Depth and stereo
GoMVS: Geometrically Consistent Cost Aggregation for Multi-View Stereo
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GoMVS, a learning-based multi-view stereo method that propagates neighboring matching costs to the reference pixel using surface normals and plane-induced depth ratios before convolutional aggregation.
Problem
Cost aggregation in learning-based multi-view stereo suffers from local geometric inconsistency: adjacent pixels' costs do not peak at the reference pixel's true depth, and prior fixes operate only on 2D depth outputs.
Approach
Assume local planarity with surface normals to compute depth-ratio correspondences between neighbouring and reference depth hypotheses, interpolate neighbours' costs into the reference depth space (geometrically consistent propagation), then aggregate with convolutions inside a 3D U-Net; monocular normals from Omnidata are found to work best as normal cues.
Key results
- On DTU, GoMVS reaches overall error 0.287 and completeness 0.227, the best among compared methods (MVSFormer 0.289 overall).
- GoMVS ranks 1st on the Tanks and Temples advanced benchmark and achieves the best F-score on intermediate (mean 66.44) and advanced (mean 44.82) sets.
- On ETH3D test split, GoMVS achieves 85.91 F-score, outperforming the prior best (EPNet 83.72).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use geometrically consistent cost aggregation (GCA) | MVS cost aggregation · DTU | Overall: 0.315 -> 0.287 | Helped |
| use GCA | MVS cost aggregation · Tanks and Temples Intermediate | Mean F-score: 59.33 -> 60.80 | Helped |
| use GCA | MVS cost aggregation · Tanks and Temples Advanced | Mean F-score: 42.95 -> 44.82 | Helped |
| replace GCA with standard 3D convolution (ablation) | GoMVS · DTU | Overall: 0.287 -> 0.315 | Hurt |
| replace GCA with sparse convolution (ablation) | GoMVS · DTU | Overall: 0.287 -> 0.311 | Hurt |
| replace GCA with spatial deformable aggregation (ablation) | GoMVS · DTU | Overall: 0.287 -> 0.310 | Hurt |
| replace cost aggregation with depth kernel regression (ablation) | GoMVS · DTU | Overall: 0.287 -> 0.316 | Hurt |
| expand standard Conv3D depth kernel from 3 to 5 (ablation) | standard Conv3D baseline · DTU | Overall: 0.315 -> 0.306 | Helped |
| expand standard Conv3D depth kernel from 3 to 7 (ablation) | standard Conv3D baseline · DTU | Overall: 0.315 -> 0.305 | Helped |
| use depth-computed normals instead of monocular normals (ablation) | GoMVS · DTU | Overall: 0.287 -> 0.297 | Hurt |
| use cost-computed normals instead of monocular normals (ablation) | GoMVS · DTU | Overall: 0.287 -> 0.300 | Hurt |
| use ground-truth normals instead of monocular normals (ablation) | GoMVS · DTU | Overall: 0.287 -> 0.248 | Helped |
Surprising findings
- Off-the-shelf monocular normals (Omnidata) outperform normals computed from the estimated depth or regressed from the cost volume, even though they lack multi-view consistency.
Limitations
- Relies on off-the-shelf monocular normal estimator whose low-resolution training requires patch-based divide-and-conquer fusion for high-resolution inputs.
- Overall DTU accuracy (0.347) is worse than several prior methods, trading accuracy for completeness.
Experimental setup
Datasets: DTU BlendedMVS
Benchmarks: DTU Tanks and Temples ETH3D
Metrics: accuracy completeness overall distance F-score precision/recall percentage at 2cm
Baselines: MVSFormer GeoMVSNet ET-MVSNet RA-MVSNet TransMVSNet UniMVSNet MVSTER Sparse convolution (NP-CVP-MVSNet) PatchmatchNet spatial deformable aggregation GeoNet depth kernel regression
Builds on: MVSNet CasMVSNet PatchmatchNet GeoNet Omnidata
Closest papers by meaning
- Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching (CVPR 2020 · similarity 0.86)
- TransMVSNet: Global Context-Aware Multi-View Stereo Network With Transformers (CVPR 2022 · similarity 0.85)
- MVS2D: Efficient Multi-View Stereo via Attention-Driven 2D Convolutions (CVPR 2022 · similarity 0.85)
- Rethinking Disparity: A Depth Range Free Multi-View Stereo Based on Disparity (AAAI 2023 · similarity 0.85)
- MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo (ICLR 2024 · similarity 0.83)
- Generalizable Novel-View Synthesis using a Stereo Camera (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.