Research map / Detection and segmentation / Open-set and few-shot detection
Matching While Perceiving: Enhance Image Feature Matching with Applicable Semantic Amalgamation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SemaGlue, a feature matcher that injects frozen SegNext semantic features into a LightGlue-style GNN matcher via a Cross-Domain Alignment block and a Semantic-Aware Fusion block.
Problem
Learned image feature matchers rely on local geometric descriptors and struggle under sparse texture, illumination changes, and occlusions; prior semantic-consistency methods use semantic priors only to pre-select matching regions.
Approach
Fuse semantic features from a frozen pre-trained segmentation encoder into every layer of the matcher after aligning semantic and geometric feature domains with a learned cross-domain correlation (CDA), then inject them via attention-based fusion (SAF), rather than using semantics only as a hard region filter.
Key results
- On MegaDepth-1500 with ALIKED and RANSAC, SemaGlue reaches 64.89 AUC@5° vs LightGlue 64.17 (per Table 2 MAGSAC++ column), with DLT AUC@5° 49.00 vs 45.03.
- On HPatches with SP, SemaGlue achieves 34.2% Acc@1px vs LightGlue 33.6%, and 95.9% Acc@3px vs 94.6%.
- On ScanNet zero-shot, SemaGlue gets 15.10 AUC@5° with RANSAC vs LightGlue 14.23 (about 6.1% relative gain).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SemaGlue instead of LightGlue with SP features | Image feature matching · Hpatches | accuracy @1px: 33.6 -> 34.2 | Helped |
| Use SemaGlue instead of LightGlue with SP features | Image feature matching · MegaDepth-1500 | AUC RANSAC @5°: 47.78 -> 49.41 | Helped |
| Use SemaGlue instead of LightGlue with ALIKED features | Image feature matching · YFCC100M | AUC RANSAC @5°: 43.89 -> 44.65 | Helped |
| Use SemaGlue instead of LightGlue with SP features | Visual localization · Aachen Day-Night v1.0 Night | pose recall (0.25m, 2°): 84.7 -> 86.7 | Helped |
| Add semantic features and SAF without CDA (ablation) | SemaGlue · MegaDepth-1500 | AUC RANSAC @5°: 47.34 -> 47.35 | No clear effect |
| Add inherent CDA blocks to semantic fusion (ablation) | SemaGlue · MegaDepth-1500 | AUC RANSAC @5°: 47.35 -> 48.00 | Helped |
| Add the extra CDA block before the first layer (ablation) | SemaGlue · MegaDepth-1500 | AUC RANSAC @5°: 48.00 -> 49.41 | Helped |
| Add SAF blocks (ablation) | SemaGlue · MegaDepth-1500 | AUC RANSAC @5°: 47.89 -> 49.41 | Helped |
Limitations
- Relies on a frozen pre-trained segmentation model whose semantic priors may be imprecise.
- Reported gains over LightGlue are modest in several settings.
Experimental setup
Datasets: Oxford and Paris (Radenovic et al. 2018) MegaDepth HPatches MegaDepth-1500 YFCC100M Aachen Day-Night v1.0 InLoc ScanNet
Benchmarks: HPatches MegaDepth-1500 YFCC100M Aachen Day-Night v1.0 InLoc ScanNet
Metrics: Accuracy at 1px/3px AUC of reprojection error AUC@5°/10° pose error pose recall at distance/orientation thresholds runtime and memory
Baselines: MNN SuperGlue SGMNet ResMatch IMP LightGlue OmniGlue LoFTR COTR PDC-Net+
Builds on: SuperGlue LightGlue SegNext OmniGlue MESA
Closest papers by meaning
- SemStereo: Semantic-Constrained Stereo Matching Network for Remote Sensing (AAAI 2025 · similarity 0.85)
- SemAlign3D: Semantic Correspondence between RGB-Images through Aligning 3D Object-Class Representations (CVPR 2025 · similarity 0.85)
- OmniGlue: Generalizable Feature Matching with Foundation Model Guidance (CVPR 2024 · similarity 0.85)
- Pixel-level Semantic Correspondence through Layout-aware Representation Learning and Multi-scale Matching Integration (CVPR 2024 · similarity 0.85)
- MATCHA: Towards Matching Anything (CVPR 2025 · similarity 0.84)
- CALICO: Part-Focused Semantic Co-Segmentation with Large Vision-Language Models (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.