Research map / Detection and segmentation / Open-set and few-shot detection
MESA: Matching Everything by Segmenting Anything
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
MESA reduces feature matching redundancy by establishing area matches from SAM segmentation, modeled with a multi-relational Area Graph and solved via graph cut and global energy refinement before point matching.
Problem
Feature matching between images wastes computation and suffers errors from matching redundancy; prior area-to-point matching relied on semantic segmentation, which is limited by labeling errors and closed-set labels.
Approach
Use class-agnostic SAM segments as areas, connect them in a multi-relational Area Graph (adjacency and inclusion edges with graph completion for scale hierarchy), cast area matching as energy minimization over an MRF/Bayesian network solved by graph cut, with a learned patch-activity area similarity and a global matching energy refinement.
Key results
- On ScanNet1500 pose estimation, MESA with DKM reaches AUC@5 of 33.4 (+13.61% over DKM baseline), a new state-of-the-art in indoor pose estimation.
- Area matching on ScanNet1500 shows MESA AOR of 67.98 versus 60.59 for SGAM with GAM (phi=1.0).
- On MegaDepth1500, MESA with DKM reaches AUC@5 of 61.1 (+1.16%), and MESA with QuadT improves AUC@5 by 6.04% over the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MESA area matching before DKM | DKM · ScanNet1500 pose estimation | AUC@5°: 29.4 -> 33.4+13.61% | Helped |
| Apply MESA area matching before DKM | DKM · ScanNet1500 pose estimation | AUC@10°: 50.7 -> 55.0+8.48% | Helped |
| Apply MESA area matching before DKM | DKM · MegaDepth1500 pose estimation | AUC@5°: 60.4 -> 61.1+1.16% | Helped |
| Apply MESA area matching before DKM | DKM · MegaDepth1500 pose estimation | AUC@20°: 85.1 -> 85.6+0.59% | Helped |
| Replace Area Graph matching with dense area similarity comparison (ablation) | MESA ASpan · ScanNet1500 area matching | AOR: 72.75 -> 69.23 | Hurt |
| Replace learned patch-level area similarity with descriptor similarity (ablation) | MESA ASpan · ScanNet1500 area matching | AMP@0.6: 89.09 -> 62.91 | Hurt |
| Replace SAM segmentation with SEEM segmentation (ablation) | MESA ASpan · ScanNet1500 pose estimation | PoseAUC@5°: 27.50 -> 26.18 | Hurt |
| Replace global matching energy refinement with Eself-only matching (ablation) | MESA ASpan · ScanNet1500 area matching | AOR: 72.75 -> 70.98 | Hurt |
Surprising findings
- Gains for semi-dense matchers were comparable to SGAM on indoor pose estimation, while the dense DKM gained far more than SGAM.
- On outdoor MegaDepth, improvement for the dense DKM was small (+1.16% AUC@5) because repetitive patterns make area matching hard.
Limitations
- Gains on dense DKM are limited in outdoor scenes with repetitive patterns.
- Two accuracy declines on KITTI360 for QuadT, attributed to its quadtree attention splitting semantic areas.
- Area matching is roughly 10x slower with dense comparison baseline; the pipeline adds computational overhead from SAM and graph processing.
Experimental setup
Datasets: ScanNet MegaDepth KITTI360 ScanNet1500 MegaDepth1500
Benchmarks: ScanNet1500 MegaDepth1500 KITTI360
Metrics: AOR AMP@0.6 AMP@0.7 AMP@0.8 AreaNum AUC@5 AUC@10 AUC@20 RPE rotational error RPE translation error
Baselines: SGAM DKM LoFTR ASpan QuadT OETR TopicFM PATS CasMTR LightGlue
Builds on: SGAM Area to Point Matching SAM graph cut
Closest papers by meaning
- MATCHA: Towards Matching Anything (CVPR 2025 · similarity 0.84)
- OmniGlue: Generalizable Feature Matching with Foundation Model Guidance (CVPR 2024 · similarity 0.83)
- SegMASt3R: Geometry Grounded Segment Matching (NeurIPS 2025 · similarity 0.83)
- RUBIK: A Structured Benchmark for Image Matching across Geometric Challenges (CVPR 2025 · similarity 0.82)
- Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.82)
- GIM: Learning Generalizable Image Matcher From Internet Videos (ICLR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.