Research map / Detection and segmentation / Open-set and few-shot detection
Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RIM, a training-free open-vocabulary segmentation framework that builds image-modal category references with Stable Diffusion and SAM and classifies regions via relation-aware ranking-distribution matching in DINOv2 space.
Problem
Open-vocabulary semantic segmentation suffers from false matches between region features and category labels due to the gap between textual (CLIP) and visual features in region classification.
Approach
Replace cross-modal text-to-image matching with image-to-image matching: category reference features are generated from Stable Diffusion images segmented by SAM and compared with SAM region features in DINOv2 space, using ranking-distribution similarity over category agents rather than individual cosine comparisons.
Key results
- Achieves 77.8 mIoU on PASCAL VOC, 34.3 on PASCAL Context and 44.5 on COCO Object, zero-shot training-free.
- Outperforms prior training-free OVDiff by 10.7 (VOC), 4.2 (Context), 9.7 (COCO) mIoU; reports a lead of more than 10% on PASCAL VOC.
- Image-to-image DINOv2 matching beats CLIP cross-modal matching by 12.0 mIoU on COCO Object in the ablation (26.7 to 38.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CLIP image-to-text matching with DINOv2 image-to-image matching (ablation) | RIM region classification on COCO Object · COCO Object | mIoU: 26.7 -> 38.7 | Helped |
| add foreground masks to synthesized image references (ablation) | RIM reference construction · COCO Object | mIoU: 38.7 -> 41.8 | Helped |
| add relation-aware matching based on ranking distributions (ablation) | RIM region classification · COCO Object | mIoU: 41.8 -> 43.1 | Helped |
| add subcategory reference features (ablation) | RIM relation-aware matching · COCO Object | mIoU: 43.1% to 44.5% | Helped |
| replace DINOv2 with CLIP as the image feature extractor (ablation) | RIM · COCO Object | mIoU: 44.5 -> 42.0 | Hurt |
| replace DINOv2 with Stable Diffusion as the image feature extractor (ablation) | RIM · COCO Object | mIoU: 44.5 -> 43.1 | Hurt |
| replace DINOv2 with MAE as the image feature extractor (ablation) | RIM · COCO Object | mIoU: 44.5 -> 39.5 | Hurt |
| add mask proposal generation instead of direct pixel classification (ablation) | RIM on PASCAL VOC · PASCAL VOC | mIoU: 42.0 -> 76.7 | Helped |
| replace MaskFormer proposals with SAM proposals (ablation) | RIM on PASCAL VOC · PASCAL VOC | mIoU: 76.7 -> 77.8 | Helped |
| select top-N similar category agents instead of selecting agents randomly (ablation) | RIM relation-aware matching · COCO Object | mIoU: 43.8 -> 44.5 | Helped |
| vary the number of subcategory references (ablation) | RIM relation-aware matching · COCO Object | mIoU: 44.1, 44.3, 44.4, 44.5, 44.5, 44.3 | Mixed |
| vary the cross-attention map binarization threshold (ablation) | RIM reference construction · COCO Object | mIoU: 42.05, 42.55, 43.05, 44.05, 44.55 | Mixed |
Surprising findings
- A simple cosine-similarity intra-modal matching with DINOv2 already yields large gains over CLIP cross-modal matching.
Limitations
- Relies on multiple large foundation models (SD, SAM, DINOv2) at inference, increasing computational cost.
- Performance depends on quality of synthesized reference images and cross-attention localization.
Experimental setup
Datasets: synthetic images generated by Stable Diffusion v1.4 per category
Benchmarks: PASCAL VOC 2012 PASCAL Context COCO Object
Metrics: mIoU
Baselines: MaskCLIP GroupViT SegCLIP SimSeg DiffSeg OVDiff ZeroSeg TCL CLIPpy OVSegmentor
Builds on: Stable Diffusion Segment Anything Model DINOv2 MaskCLIP OVDiff
Closest papers by meaning
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.92)
- Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation (CVPR 2024 · similarity 0.90)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.89)
- OVMR: Open-Vocabulary Recognition with Multi-Modal References (CVPR 2024 · similarity 0.88)
- Relationship Prompt Learning is Enough for Open-Vocabulary Semantic Segmentation (NeurIPS 2024 · similarity 0.88)
- Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.