Research map / 3D vision / 3D generation
SegMASt3R: Geometry Grounded Segment Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts the 3D foundation model MASt3R for wide-baseline segment matching by adding a lightweight segment-feature head trained with a Sinkhorn/dustbin matcher, outperforming local feature matchers and SAM2 propagation on ScanNet++, Replica and MapFree.
Problem
Segment matching across wide-baseline image pairs (up to 180° rotation) fails with 2D-trained features, which mismatch repetitive patterns and fail on extreme viewpoint changes.
Approach
Reuse MASt3R's geometry-aware patch features and add a segment-feature head that pools upsampled pixel descriptors inside each mask, then matches segments with a learnable-dustbin Sinkhorn layer trained with a SuperGlue-style cross-entropy loss.
Key results
- On ScanNet++ 135-180° pose bin, SegMASt3R reaches 83.6 AUPRC vs 45.4 for MASt3R LFM and 17.0 for SAM2.
- On Replica (zero-shot from ScanNet++), SegMASt3R achieves 68.4 AUPRC in the 135-180° bin vs 32.5 for MASt3R LFM.
- In navigation with S_s=16, S_rho=0.25, SegMASt3R improves SPL from 36.34 to 63.60 over RoboHop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn segment-level descriptors from MASt3R geometry-aware features rather than the strongest baseline representations | Wide-baseline segment matching · ScanNet++ 135°–180° | AUPRC: 45.4 -> 83.6 | Helped |
| Use the trained segment matcher rather than MASt3R local feature matching | MASt3R · ScanNet++ 90°–135° | AUPRC: 52.9 -> 88.0 | Helped |
| Use the trained segment matcher rather than MASt3R local feature matching | MASt3R · Replica 135°–180° | AUPRC: 32.5 -> 68.4 | Helped |
| Recalibrate the learnable dustbin parameter on MapFree (ablation) | SegMASt3R (SPP) · MapFree validation | Overall IoU: 75.2 -> 88.7 | Helped |
| Train on MapFree instead of using the indoor-trained model (ablation) | SegMASt3R · MapFree validation | Overall IoU: 75.2 -> 93.7 | Helped |
| Replace MASt3R cross-attention features with DINOv2 features (ablation) | SegMASt3R · ScanNet++ 0°–45° | AUPRC: 92.8 -> 64.7 | Hurt |
| Replace MASt3R cross-attention features with CroCo features (ablation) | SegMASt3R · ScanNet++ 0°–45° | AUPRC: 92.8 -> 73.4 | Hurt |
| Replace MASt3R cross-attention features with DINOv2 features (ablation) | SegMASt3R · ScanNet++ 135°–180° | AUPRC: 83.6 -> 36.8 | Hurt |
| Replace MASt3R cross-attention features with CroCo features (ablation) | SegMASt3R · ScanNet++ 135°–180° | AUPRC: 83.6 -> 38.5 | Hurt |
| Replace RoboHop's SuperPoint+LightGlue segment matching with SegMASt3R | RoboHop · ImageInstanceNav validation, Ss=16, Sρ=0.25 | SPL: 36.34 -> 63.60 | Helped |
| Use SegMASt3R for pairwise object correspondence in place of ConceptGraphs matching | Replica instance mapping · Replica room0 with ground-truth masks | AP: 44.89 -> 71.02 | Helped |
| Use SegMASt3R rather than MASt3R vanilla features with noisy FastSAM masks | ScanNet++ segment matching · ScanNet++ noisy FastSAM masks, IoU > 0.25 | AUPRC: 50.7 -> 84.2 | Helped |
Surprising findings
- Plugging MASt3R's dense features into a segment-level pooling drops AUPRC sharply (e.g., 52.9 to 41.4 in the 90-135° bin), suggesting pixel-level distinctiveness does not transfer to instance level.
- A 2D-only DINOv2 or CroCo segment-feature baseline performs far below the 3D-grounded model, indicating geometric context is essential.
Limitations
- Relies on 3D instance-segmentation ground truth for training; MapFree variant uses SAM2 pseudo-labels that may propagate errors.
- Indoor-trained model regresses on outdoor MapFree without domain-specific retraining or dustbin recalibration.
Experimental setup
Datasets: ScanNet++ Replica MapFree
Benchmarks: ScanNet++ Replica MapFree ImageInstanceNav HM3Dv0.2
Metrics: AUPRC Recall@1 Recall@5 AP AP@50 SPL SSPL IoU
Baselines: SP-LG GiM-DKM RoMA MASt3R SAM2 DINOv2 SegVLAD ConceptGraphs RoboHop
Builds on: MASt3R SuperGlue CroCo SAM2
Closest papers by meaning
- CoMatcher: Multi-View Collaborative Feature Matching (CVPR 2025 · similarity 0.86)
- SegGraph: Leveraging Graphs of SAM Segments for Few-Shot 3D Part Segmentation (NeurIPS 2025 · similarity 0.85)
- HomoMatcher: Achieving Dense Feature Matching with Semi-Dense Efficiency by Homography Estimation (AAAI 2025 · similarity 0.85)
- Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching (ICLR 2024 · similarity 0.85)
- AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation (ICLR 2024 · similarity 0.85)
- Partial-to-Partial Shape Matching with Geometric Consistency (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.