Research map / Vision-language and multimodal / CLIP and zero-shot
OVMR: Open-Vocabulary Recognition with Multi-Modal References
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OVMR, a plug-and-play open-vocabulary recognition method that fuses text descriptions and exemplar images via a lightweight pre-trained visual token generator and a parameter-free preference-based fusion of text, vision, and multi-modal classifiers.
Problem
Open-vocabulary recognition of novel categories where text descriptions can be ambiguous and exemplar images vary in quality, while fine-tuning per dataset is time-consuming and harms generalization.
Approach
Generate a multi-modal classifier by learning visual tokens from exemplar images and letting the CLIP text encoder fuse them with text tokens, then fuse text-, vision-, and multi-modal classifiers using per-category preferences measured by F1 on the exemplar images themselves (test-time, no trainable fusion parameters).
Key results
- On 11 classification datasets in the prompt-learning setup, OVMR (training-free) reaches 82.34% average accuracy, comparable to CoOp's 82.69%, and OVMR with CoOp prompt tokens reaches 83.72%.
- On LVIS with ResNet-50, OVMR achieves 21.2 mask APr versus 19.3 for MM-OVOD; with Swin-B it reaches 34.4 mask APr, above Detic's 33.8.
- Preference-based F1 fusion improves average accuracy from 80.99% (multi-modal classifier only) to 82.34%, and the three-classifier fusion gives a 15.54-point gain over the text-only baseline (66.80%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multi-modal exemplar cues and preference-based fusion versus MM-OVOD | OVMR with CenterNet2, ResNet-50 · LVIS rare classes | mask APr: 19.3 -> 21.2 | Helped |
| use OVMR rather than Detic | CenterNet2 with Swin-B on LVIS · LVIS rare classes | mask APr: 33.8 -> 34.4 | Helped |
| add only the vision-based classifier instead of text-based classifier only (ablation) | OVMR · 11 classification datasets | average accuracy: 66.80 -> 80.28 | Helped |
| add a multi-modal classifier instead of text-based classifier only (ablation) | OVMR · 11 classification datasets | average accuracy: 66.80 -> 80.99 | Helped |
| remove the pre-training strategy and dropout operations (ablation) | OVMR multi-modal classifier · 11 classification datasets | average accuracy: 80.99 -> 77.86 | Hurt |
| fuse text-based and vision-based classifiers (ablation) | OVMR · 11 classification datasets | average accuracy: 66.80 -> 81.71 | Helped |
| fuse text-based and multi-modal classifiers (ablation) | OVMR · 11 classification datasets | average accuracy: 66.80 -> 81.30 | Helped |
| fuse vision-based and multi-modal classifiers (ablation) | OVMR · 11 classification datasets | average accuracy: 80.99 -> 81.96 | Helped |
| fuse all three classifiers using preference weights (ablation) | OVMR · 11 classification datasets | average accuracy: 81.96 -> 82.34 | Helped |
| replace preference-based fusion with equal-weight mean fusion (ablation) | OVMR · 11 classification datasets | average accuracy: 82.34 -> 81.45 | Hurt |
| use F1 instead of precision for preference generation (ablation) | OVMR · 11 classification datasets | average accuracy: 82.03 -> 82.34 | Helped |
| use F1 instead of recall for preference generation (ablation) | OVMR · 11 classification datasets | average accuracy: 81.70 -> 82.34 | Helped |
Surprising findings
- Using only internet-crawled exemplar images on ImageNet still yields a notable gain (74.56% vs 66.80% text-only zero-shot).
- Text-guided dynamic fusion (MQ-Det) performed worst, at 62.60% multi-modal accuracy, below the text-only baseline.
Limitations
- Preference-based fusion degrades slightly in the 2-shot setting because few exemplars cannot reliably estimate per-category preferences.
- Detection experiments require a memory-efficient re-implementation of Detic with limited ground-truth boxes per image (10), so comparisons depend on this reproduction.
- Classification results are reported on base classes (first half of each dataset's categories).
Experimental setup
Datasets: ImageNet21k-OVR ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT LVIS
Metrics: accuracy mask APr mask AP F1
Baselines: CLIP CoOp CoCoOp MaPLe MAML MatchingNet LR PN CEPA CC URL ViLD-Ens
Builds on: CLIP CoOp Detic MM-OVOD MQ-Det
Closest papers by meaning
- LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors (ICLR 2024 · similarity 0.88)
- Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.88)
- LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models (CVPR 2025 · similarity 0.88)
- Comprehensive Multi-Modal Prototypes Are Simple and Effective Classifiers for Vast-Vocabulary Object Detection (AAAI 2025 · similarity 0.88)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.88)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.