Research map / 3D vision / Depth and stereo
RRT-MVS: Recurrent Regularization Transformer for Multi-View Stereo
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RRT-MVS, a cascade multi-view stereo network whose Recurrent Regularization Transformer regularizes the cost volume per depth hypothesis plane with recurrent self-attention, depth residual attention, and a positional adapter.
Problem
Cost volume regularization in learning-based multi-view stereo: CNNs have limited receptive fields, while Transformers that merge depth and spatial dimensions struggle with depth discontinuities at object boundaries and backgrounds.
Approach
Split the cost volume into 2D cost maps along the depth dimension, apply window-based self-attention plus convolutional channel/spatial attention within each map, link maps recurrently across depth, and add depth residual attention and a positional adapter to model depth correlations and 3D position separately from the spatial dimensions.
Key results
- On DTU test set, RRT-MVS reaches overall error 0.285 mm, the lowest among the compared methods (GoMVS 0.287).
- On Tanks-and-Temples, RRT-MVS achieves mean F-score 68.16 on intermediate and 43.29 on advanced subsets, ranked first among published methods.
- Compared with GoMVS, RRT-MVS reduces additional GPU memory by about 97% (+266 MB vs +9114 MB) at the same overall gain of -0.028 mm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RRT cost-volume regularization | RRT-MVS · DTU test | Overall error (mm): 0.287 -> 0.285 | Helped |
| Add R-SA (ablation) | RRT-MVS · DTU test | Overall error (mm): 0.318 -> 0.299 | Helped |
| Add DRA to R-SA (ablation) | RRT-MVS · DTU test | MAE: 6.35 -> 5.71 | Helped |
| Add PA to R-SA (ablation) | RRT-MVS · DTU test | Overall error (mm): 0.299 -> 0.295 | Helped |
| Use R-SA, DRA, and PA together (ablation) | RRT-MVS · DTU test | Overall error (mm): 0.318 -> 0.290 | Helped |
| Use RRT instead of 3D CNN alone (ablation) | RRT-MVS · DTU test | Overall error (mm): 0.318 -> 0.311 | Helped |
| Combine RRT with 3D CNN (ablation) | RRT-MVS · DTU test | Overall error (mm): 0.311 -> 0.290 | Helped |
| Add RRT to CasMVSNet (ablation) | CasMVSNet · DTU test | Overall error (mm): 0.355 -> 0.326 | Helped |
| Add RRT to MVSTER (ablation) | MVSTER · DTU test | Overall error (mm): 0.313 -> 0.301 | Helped |
| Use RRT-MVS | MVS methods evaluated on Tanks-and-Temples intermediate · Tanks-and-Temples intermediate | Mean F-score (%): 67.18 -> 68.16 | Helped |
| Use RRT-MVS | MVS methods evaluated on Tanks-and-Temples advanced · Tanks-and-Temples advanced | Mean F-score (%): 43.07 -> 43.29 | Helped |
Surprising findings
- Adding RRT to the 3D CNN regularization improves depth map quality strongly but only modestly improves point cloud quality, which the authors attribute to RRT's lack of local detail.
Limitations
- RRT lacks local detail understanding, so it must be combined with a 3D CNN to get the best point cloud quality.
- Gains on DTU are small in absolute terms (overall 0.285 vs 0.287 for GoMVS).
Experimental setup
Datasets: DTU BlendedMVS Tanks-and-Temples
Benchmarks: DTU Tanks-and-Temples
Metrics: accuracy completeness overall distance MAE depth error ratio e2/e4/e8 F-score GPU memory
Baselines: GoMVS GeoMVSNet ET-MVSNet MVSFormer++ CostFormer WT-MVSNet DMVSNet MVSTER CasMVSNet
Builds on: MVSNet CasMVSNet MVSTER Swin Transformer CBAM
Closest papers by meaning
- Recurrent MVSNet for High-Resolution Multi-View Stereo Depth Inference (CVPR 2019 · similarity 0.90)
- RIAV-MVS: Recurrent-Indexing an Asymmetric Volume for Multi-View Stereo (CVPR 2023 · similarity 0.90)
- MVS2D: Efficient Multi-View Stereo via Attention-Driven 2D Convolutions (CVPR 2022 · similarity 0.89)
- Fast-MVSNet: Sparse-to-Dense Multi-View Stereo With Learned Propagation and Gauss-Newton Refinement (CVPR 2020 · similarity 0.89)
- MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo (ICLR 2024 · similarity 0.89)
- A Global Depth-Range-Free Multi-View Stereo Transformer Network with Pose Embedding (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.