Research map / Detection and segmentation / Open-set and few-shot detection
VL-SAM-V2: Open-World Object Detection with General and Specific Query Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VL-SAM-V2, an open-world detector that fuses VL-SAM point-prompted general queries with open-set (LLMDet) specific queries, switchable between open-set and open-ended modes, reaching SOTA on LVIS.
Problem
Open-set detectors need human-provided category lists and miss rare objects, while open-ended detectors discover objects without category input but perform far worse than open-set models, especially on frequent classes.
Approach
Fuse queries from an open-set detector (specific queries) with VL-SAM point-prompt-derived queries (general queries) via a lightweight general-and-specific query fusion self-attention, using ranked learnable queries matched to score-sorted points and denoising points for training; the category list can be emptied to run in open-ended mode.
Key results
- Open-set: with Swin-T, VL-SAM-V2 reaches 45.7 AP and 41.2 AP_r on LVIS minival (vs. LLMDet 44.7 AP, 37.3 AP_r); with Swin-L 51.7 AP on minival and 42.5 AP on LVIS val.
- Open-ended: with Swin-L, VL-SAM-V2 reaches 31.8 AP and 30.5 AP_rare on LVIS minival, above GenerateU (27.9 AP, 22.3 AP_r) and VL-SAM (25.3 AP, 23.4 AP_r).
- Ablation: GS fusion, ranked learnable queries and denoising points add 0.5, 0.2 and 0.3 AP respectively over the LLMDet baseline (45.7 AP total with all three).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add general and specific query fusion (ablation) | LLMDet with Swin-T · LVIS minival | AP: 44.7 -> 45.2 | Helped |
| add general and specific query fusion (ablation) | LLMDet with Swin-T · LVIS minival | AP_r: 37.3 -> 39.1 | Helped |
| add ranked learnable queries instead of a single learnable query per point prompt (ablation) | LLMDet with GS Fusion and Swin-T · LVIS minival | AP_r: +1.1 AP_r | Helped |
| add denoising points during training (ablation) | LLMDet with GS Fusion and ranked learnable queries, Swin-T · LVIS minival | AP_r: +1.0 AP_r | Helped |
| use VL-SAM-V2 instead of LLMDet | Swin-T open-set detection · LVIS minival | AP: 44.7 -> 45.7 | Helped |
| use VL-SAM-V2 instead of LLMDet | Swin-T open-set detection · LVIS minival | AP_r: 37.3 -> 41.2 | Helped |
| use VL-SAM-V2 instead of LLMDet | Swin-L open-set detection · LVIS minival | AP: 51.1 -> 51.7 | Helped |
| use VL-SAM-V2 instead of LLMDet | Swin-L open-set detection · LVIS minival | AP_r: 45.1 -> 47.2 | Helped |
| use VL-SAM-V2 instead of GenerateU | Swin-T open-ended detection · LVIS minival | AP: +2.7 AP | Helped |
| use VL-SAM-V2 instead of GenerateU | Swin-T open-ended detection · LVIS minival | AP_rare: +9.8 AP_r | Helped |
| use VL-SAM-V2 instead of VL-SAM | open-ended detection with Swin-L · LVIS minival | AP_rare: 23.4 AP_r to 30.5 AP_r | Helped |
| combine VL-SAM-V2 with SAM instead of using VL-SAM with SAM | open-ended instance segmentation · LVIS minival | mask AP: 23.9 -> 28.7 | Helped |
Surprising findings
- DetCLIPv3 achieves higher AP_r and AP_c than VL-SAM-V2 on LVIS minival with Swin-L, which the authors attribute to more balanced data and noun concept corpora.
Limitations
- Inherits hallucination, incorrect responses and low inference speed of the underlying vision-language models.
- Requires an additional SAM to produce instance segmentation masks.
Experimental setup
Datasets: GroundingCap-1M LVIS CODA
Benchmarks: LVIS minival LVIS val CODA
Metrics: fixed AP AP_rare AP_common AP_frequent mask AP
Baselines: GLIP GLIPv2 Grounding-DINO DetCLIP DetCLIPv2 DetCLIPv3 YOLO-World OV-DINO LLMDet GenerateU VL-SAM CapDet
Builds on: VL-SAM LLMDet DINO GenerateU Grounded-SAM
Closest papers by meaning
- Training-Free Open-Ended Object Detection and Segmentation via Attention as Prompts (NeurIPS 2024 · similarity 0.91)
- OW-OVD: Unified Open World and Open Vocabulary Object Detection (CVPR 2025 · similarity 0.89)
- Open-World Objectness Modeling Unifies Novel Object Detection (CVPR 2025 · similarity 0.88)
- Open-Det: An Efficient Learning Framework for Open-Ended Detection (ICML 2025 · similarity 0.88)
- Exploring Orthogonality in Open World Object Detection (CVPR 2024 · similarity 0.88)
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.