Research map / Detection and segmentation / Open-set and few-shot detection
Comprehensive Multi-Modal Prototypes Are Simple and Effective Classifiers for Vast-Vocabulary Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prova, a multi-modal prototype classifier that initializes alignment classifiers with LLM-generated textual description embeddings and CLIP visual exemplar embeddings, improving detectors on the 13K-category V3Det dataset.
Problem
Open-world and closed-set detectors using coarse class-name alignment classifiers degrade badly as the category vocabulary scales to real-world size (e.g., V3Det with 13,204 categories), with semantic confusion and ambiguity between fine-grained classes.
Approach
Replace coarse class-name embeddings in alignment classifiers with multi-modal prototypes: detailed description embeddings (from a long-text CLIP) and averaged visual exemplar embeddings (from CLIP-ViT-L), ensembled by averaging logits with the conventional classifier in the supervised setting, adding only projection layers.
Key results
- Faster R-CNN, FCOS, and DINO with Prova gain +3.3, +6.2, and +2.9 AP on V3Det supervised (ResNet-50).
- Open-vocabulary DINO + Prova reaches 32.8 base AP and 11.0 novel AP on V3Det with ImageNet-22K data, +2.6 base and +4.3 novel over prior methods.
- DINO + Prova with Swin-Base (213.4M params) reaches 50.3 AP on V3Det, outperforming Cascade Mask R-CNN with EVA-ViT-G (1.1B) at 49.4 AP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Prova multi-modal prototype classifier (projection layers + prototype alignment) | Faster R-CNN · V3Det val supervised, ResNet-50 | AP: +3.3 AP | Helped |
| add Prova multi-modal prototype classifier | FCOS · V3Det val supervised, ResNet-50 | AP: +6.2 AP | Helped |
| add Prova multi-modal prototype classifier | DINO · V3Det val supervised, ResNet-50 and Swin-Base | AP: +2.9 AP (R50); +2.5 AP (Swin-Base) | Helped |
| replace conventional classifier with description-based textual prototype classifier only (ablation) | DINO (12-epoch ablation) · V3Det val supervised | AP: 16.0 vs 24.5 conventional | Hurt |
| use class-name alignment classifier instead of description or visual prototypes (ablation) | DINO (12-epoch ablation) · V3Det val supervised | AP: 10.8 vs 24.5 conventional | Hurt |
| use visual prototype classifier alone instead of conventional classifier (ablation) | DINO (12-epoch ablation) · V3Det val supervised | AP: +3.6 AP | Helped |
| ensemble description and visual prototype classifiers (without conventional classifier) (ablation) | DINO (12-epoch ablation) · V3Det val supervised | AP: +1.3 AP over visual alone | Helped |
| add conventional classifier to ensemble of description and visual prototypes (ablation) | DINO (12-epoch ablation) · V3Det val supervised | AP: +4.6 AP | Helped |
| use more reference images per category for visual prototypes (1 to 100) (ablation) | Visual prototypes in Prova · V3Det val supervised, 12-epoch ablation | AP: 21.3 -> 26.6 -> 28.1 | Helped |
| use LongCLIP-Large instead of CLIP-ViT-Large for textual prototype encoder (ablation) | Textual prototypes · V3Det val supervised, 12-epoch ablation | AP: 16.0 vs 11.9 | Helped |
| use CLIP-ViT-Large instead of SigLIP-Large or DINOv2-Large for visual prototype encoder (ablation) | Visual prototypes · V3Det val supervised, 12-epoch ablation | AP: 28.1 vs 27.5 (SigLIP) and 25.5 (DINOv2) | Helped |
| add Prova prototype classifier, no extra data | DINO (open-vocabulary) · V3Det val open-vocab, base/novel | AP_novel: +6.5 AP_novel over CenterNet2/Detic baseline | Helped |
Surprising findings
- Class-name alignment classifier performs worse than the conventional closed-set classifier on V3Det (10.8 AP vs 24.5 AP in the ablation).
- Visual prototypes alone outperform the description-based textual prototypes (28.1 vs 16.0 AP).
Limitations
- Relies on pre-collected detailed descriptions from the V3Det Challenge generated by GPT-4V, which may not be available for other vocabularies.
- Open-vocabulary novel-class performance depends on reference images from ImageNet-22K and V3Det exemplars, limiting fully novel-data settings.
- Gains on LVIS are reported with only class-name textual prototypes and visual prototypes, without the description prototypes.
Experimental setup
Datasets: V3Det ImageNet-22K ImageNet-1K LVIS Object365 V3Det Challenge 2024 descriptions
Benchmarks: V3Det OV-LVIS LVIS
Metrics: AP AP50 AP75 AP_base AP_novel AP_final AP_r
Baselines: Faster R-CNN FCOS DINO RegionCLIP CenterNet2 DST-Det Detic Cascade Mask R-CNN CORA CFM-ViT
Builds on: DINO Faster R-CNN FCOS CLIP Detic
Closest papers by meaning
- LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors (ICLR 2024 · similarity 0.89)
- OVMR: Open-Vocabulary Recognition with Multi-Modal References (CVPR 2024 · similarity 0.88)
- Open-Det: An Efficient Learning Framework for Open-Ended Detection (ICML 2025 · similarity 0.88)
- DitHub: A Modular Framework for Incremental Open-Vocabulary Object Detection (NeurIPS 2025 · similarity 0.88)
- Exploring Region-Word Alignment in Built-in Detector for Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.87)
- How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.