Research map / Vision-language and multimodal / CLIP and zero-shot
Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Matcher is a training-free framework that segments anything from one in-context example by matching DINOv2 patch features to guide SAM-generated mask proposals, filtered with optimal-transport and purity/coverage metrics.
Problem
Vision foundation models need task-specific structures and fine-tuning for perception tasks; combining off-the-shelf models for diverse segmentation tasks (semantic, part, video) without training is non-trivial since SAM lacks semantic labels and produces ambiguous masks.
Approach
Use DINOv2 dense patch correspondence from a single reference image-mask pair to produce coarse matched points, refine them with bidirectional matching and clustering-based prompt sampling for SAM, and select masks via instance-level matching (EMD plus purity and coverage scores).
Key results
- Matcher achieves 52.7% mean mIoU one-shot on COCO-20i, surpassing the best specialist model by 1.6%, with no training.
- Matcher achieves 33.0% mIoU one-shot on LVIS-92i, outperforming SegGPT by 14.4%.
- Matcher reaches 79.5 J&F on DAVIS 2017 val, outperforming SegGPT (75.6) and PerSAM-F (71.9) without video training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add instance-level matching (ablation) | Matcher · COCO-20^i | mean mIoU: 29.0 -> 52.7 | Helped |
| add instance-level matching (ablation) | Matcher · FSS-1000 | mIoU: 76.2 -> 87.0 | Helped |
| add instance-level matching (ablation) | Matcher · DAVIS 2017 | J&F: 39.9 -> 79.5 | Helped |
| use bidirectional matching rather than forward matching alone (ablation) | Matcher · COCO-20^i; FSS-1000; DAVIS 2017 | mean mIoU; mIoU; J&F: +2.1% mean mIoU; +5.9% mIoU; +6.0% J&F | Helped |
| use reverse matching alone instead of bidirectional matching (ablation) | Matcher · COCO-20^i; FSS-1000; DAVIS 2017 | mean mIoU; mIoU; J&F: 21.4; 47.7; 41.3 | Hurt |
| combine EMD with purity and coverage instead of EMD alone (ablation) | Matcher · COCO-20^i; FSS-1000; DAVIS 2017 | mean mIoU; mIoU; J&F: 51.3 -> 52.7; 86.3 -> 87.0; 67.5 -> 79.5 | Helped |
| combine EMD with purity and coverage instead of purity and coverage alone (ablation) | Matcher · COCO-20^i; FSS-1000; DAVIS 2017 | mean mIoU; mIoU; J&F: 35.3 -> 52.7; 86.3 -> 87.0; 76.3 -> 79.5 | Helped |
| use four frames rather than one, two, or six (ablation) | Matcher video object segmentation · DAVIS 2017 | J&F: 73.5; 74.4; 79.5; 78.0 | Helped |
| replace DINOv2 with MAE or CLIP as image encoder (ablation) | Matcher · COCO-20^i; FSS-1000; DAVIS 2017 | mean mIoU; mIoU; J&F: 18.8; 71.9; 69.5 (MAE); 32.2; 77.4; 73.9 (CLIP) | Hurt |
| use global prompts instead of instance prompts (ablation) | Matcher · COCO-20^i; PACO-Part; FSS-1000 | mean mIoU; mean mIoU; mIoU: 51.7; 31.6; 87.0 | Mixed |
| use part prompts instead of instance prompts (ablation) | Matcher · COCO-20^i; PACO-Part; FSS-1000 | mean mIoU; mean mIoU; mIoU: 51.7; 30.2; 79.2 | Mixed |
| replace SAM with Semantic-SAM (ablation) | Matcher · COCO-20^i; LVIS-92^i; FSS-1000; PACO-Part | mIoU: 52.7 -> 51.1; 31.4 -> 30.1; 87.0 -> 87.5; 32.7 -> 36.0 | Mixed |
Surprising findings
- Reverse-only bipartite matching without reference-mask guidance performs very poorly (21.4 mean mIoU on COCO-20i), though it helps when combined bidirectionally.
- Semantic-SAM as segmenter performs comparably to SAM on most benchmarks, slightly better only on PACO-Part.
Limitations
- Limited instance-level matching inherited from the image encoder, restricting instance segmentation performance.
- Performance on PACO-Part remains modest (34.7 mean) and depends on the quality of foundation model matches.
- Relies on the quality of the off-the-shelf encoder; CLIP and MAE features perform much worse.
Experimental setup
Datasets: COCO-20i FSS-1000 LVIS PASCAL VOC 2010 PASCAL body part annotations PACO DAVIS 2016 DAVIS 2017
Benchmarks: COCO-20i FSS-1000 LVIS-92i PASCAL-Part PACO-Part DAVIS 2017 val DAVIS 2016 val
Metrics: mIoU mean mIoU J&F J F
Baselines: HSNet VAT FPTrans MSANet Painter SegGPT PerSAM PerSAM-F Semantic-SAM
Builds on: Segment Anything Model DINOv2 k-means++ earth mover's distance
Closest papers by meaning
- Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos (NeurIPS 2025 · similarity 0.88)
- Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.87)
- MATCHA: Towards Matching Anything (CVPR 2025 · similarity 0.87)
- OmniGlue: Generalizable Feature Matching with Foundation Model Guidance (CVPR 2024 · similarity 0.86)
- UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface (NeurIPS 2025 · similarity 0.86)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.