Research map / Detection and segmentation / Open-set and few-shot detection
Training-Free Open-Ended Object Detection and Segmentation via Attention as Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
VL-SAM is a training-free open-ended object detection and segmentation framework that links a VLM (CogVLM) to SAM via attention maps aggregated with head weights and regularized attention flow, then sampled into point prompts.
Problem
Open-world perception usually needs predefined category names at inference (open-set), which are unavailable in real scenarios; open-ended detection must discover unseen objects and their names without category inputs, while VLMs localize poorly.
Approach
Use VLM attention maps for the generated object tokens as intermediate prompts for SAM: head-aggregated attention rollout with a causal-mask regularization yields attention maps, from which positive/negative points are sampled and iteratively refined, all without any training.
Key results
- On LVIS minival, VL-SAM reaches 23.4 box AP_rare and 22.7 mask AP_rare, surpassing open-ended GenerateU (20.0 box AP_rare) without training.
- On CODA, VL-SAM achieves 40.1 mAR versus 18.4 mAR for LLaVA-Grounding, about 74.1% of the GT-box SAM oracle (54.1 mAR).
- Attention flow with regularization improves over naive attention by 6.3 mAR on CODA, and head weighting adds another 1.6 mAR (ablation reaches 10.1 mAR at that stage).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace GenerateU with VL-SAM for open-ended detection | LVIS minival rare objects · LVIS minival | box AP_rare: 20.0 -> 23.4; 3.4 AP_rare | Helped |
| Replace LLaVA-Grounding with VL-SAM | CODA object detection · CODA | mAR: 18.4 -> 40.1 | Helped |
| Add attention generation to naive attention (ablation) | VL-SAM on CODA · CODA | mAR: +7.9 mAR | Helped |
| Add positive and negative point-pair prompt generation (ablation) | VL-SAM on CODA · CODA | mAR: 10.1 -> 12.3; 2.2 mAR | Helped |
| Add iterative refinement (ablation) | VL-SAM on CODA · CODA | mAR: 12.3 -> 14.1 | Helped |
| Add multi-scale image input (ablation) | VL-SAM with attention generation, prompt generation, and iterative refinement · CODA | mAR: 14.1 -> 27.3 | Helped |
| Add question-prompt ensemble (ablation) | VL-SAM with multi-scale input · CODA | mAR: 27.3 -> 40.1 | Helped |
| Add multi-scale and question ensembles to naive attention (ablation) | Naive Attention on CODA · CODA | mAR: 2.2 -> 5.0 | No clear effect |
| Add regularization to attention flow (ablation) | VL-SAM attention generation on CODA · CODA | mAR: 0.1 -> 8.5; 6.3 mAR over naive attention | Helped |
| Add head weights to regularized attention flow (ablation) | VL-SAM attention generation on CODA · CODA | mAR: 8.5 -> 10.1; 1.6 mAR | Helped |
| Replace CogVLM with MiniGPT-4 (ablation) | VL-SAM · CODA | mAR: 40.1 -> 34.7 | Hurt |
| Replace CogVLM with LLaVA (ablation) | VL-SAM · CODA | mAR: 40.1 -> 37.2 | Hurt |
Surprising findings
- Plain attention flow without regularization almost completely fails (0.1 mAR) due to attention collapse caused by causal masks.
- Replacing CogVLM with MiniGPT-4 or LLaVA, or SAM with MobileSAM, lowers performance, though all variants still beat prior open-ended results.
Limitations
- Inherits hallucinations from the VLM, which generate wrong object tokens and attention maps.
- Low inference speed due to the VLM and SAM pipeline.
- Lags behind closed-set and some open-set methods on LVIS mask results; performance depends on VLM and SAM quality.
Experimental setup
Datasets: LVIS CODA
Benchmarks: LVIS minival CODA
Metrics: box AP rare (fixed AP) mask AP rare mAR AR50 AR75
Baselines: GenerateU LLaVA-Grounding GroundingDINO YOLO-World GLIP OWLv2 Mask R-CNN Deformable DETR RetinaNet ORE
Builds on: Segment Anything Model attention rollout PerSAM SPHINX multi-scale ensemble Tag2Text
Closest papers by meaning
- VL-SAM-V2: Open-World Object Detection with General and Specific Query Fusion (NeurIPS 2025 · similarity 0.91)
- Boosting Segment Anything Model Towards Open-Vocabulary Learning (AAAI 2025 · similarity 0.89)
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.89)
- Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm (AAAI 2025 · similarity 0.88)
- Open-Det: An Efficient Learning Framework for Open-Ended Detection (ICML 2025 · similarity 0.88)
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.