Research map / Detection and segmentation / Open-set and few-shot detection
Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Image Prompt Paradigm and MI Grounding, which uses a few automatically cropped training-set instance images as prompts (selected by a self-attention PFSM) for single-stage non-interactive open-set detection and segmentation, and introduces the ADR50K X-ray defect dataset.
Problem
Text prompts struggle to describe specialized categories, and visual prompts (points, boxes, masks) need multi-round human interaction, which blocks fully automated single-stage open-set object detection and segmentation.
Approach
Replace text and manual visual prompts with a few automatically cropped instance images per category; a prompt feature selection module based on self-attention down-weights low-quality prompts, and the fused prompt features are aligned with multi-scale image features in a GLIP-style deep-fusion detector.
Key results
- MI Grounding-D (Objects365 only) reaches 27.4 AP on LVIS-1203 versus 26.9 for GLIP L and 21.5 on ODinW-35 versus 26.1 for Grounding DINO L.
- MI Grounding-S reaches 21.0 AP on ADE20K and 46.9 AP_avg on SegInW, outperforming the best listed baselines by 4.2 and 6.3 points as claimed in the text.
- On the authors' ADR50K dataset, MI Grounding reaches 50.2 detection AP50 versus 43.4 for Grounding DINO L and 16.5 for DINOv L.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use image prompts with MI Grounding-D instead of text or visual prompts | MI Grounding-D · LVIS-1203 (out-domain) | AP_r^b: 17.1 -> 19.9 | Helped |
| use image prompts with MI Grounding-D instead of text prompts | MI Grounding-D · ODinW-35 (out-domain) | AP^b: 26.1 -> 21.5 | Hurt |
| use image prompts with MI Grounding-S instead of other prompt paradigms | MI Grounding-S · ADE20K (out-domain) | AP^m: 16.8 -> 21.0 | Helped |
| use image prompts with MI Grounding-S instead of other prompt paradigms | MI Grounding-S · COCO (in-domain) | AP^m: 53.2 -> 46.1 | Hurt |
| use image prompts instead of text or visual prompts | MI Grounding · ADR50K | Detection AP_50: 43.4 -> 50.2 | Helped |
| use image prompts instead of visual prompts | MI Grounding · ADR50K | Segmentation AP_50: 15.4 -> 50.4 | Helped |
| replace PFSM with mean pooling (ablation) | MI Grounding · COCO | AP^b: 61.7 -> 56.5 | Hurt |
| replace PFSM with a fully connected network (ablation) | MI Grounding · COCO | AP^b: 61.7 -> 54.5 | Hurt |
| replace PFSM with a convolutional neural network (ablation) | MI Grounding · COCO | AP^b: 61.7 -> 58.0 | Hurt |
| increase image prompt update frequency to once per iteration (ablation) | MI Grounding · COCO | AP^b: 57.9 -> 61.7 | Helped |
| increase image prompt quantity from 1 to 8 (ablation) | MI Grounding · COCO | AP^b: 44.6 -> 61.7 | Helped |
| increase image prompt quantity from 8 to 12 (ablation) | MI Grounding · COCO | AP^b: 61.7 -> 61.4 | No clear effect |
Surprising findings
- MI Grounding-D underperforms Grounding DINO L on ODinW-35 (21.5 vs 26.1), which is not discussed in the text.
Limitations
- Inference cost grows with the number of image prompts; gains plateau beyond 8 prompts.
- Relies on training-set instance crops as prompts, so it requires an annotated prompt library and has not been shown for novel categories without labeled instances.
Experimental setup
Datasets: Objects365 COCO LVIS ADE20K SegInW ODinW ADR50K
Benchmarks: COCO LVIS-1203 ODinW-35 ADE20K SegInW ADR50K
Metrics: AP AP50 AP_rare AP_common AP_frequent AP_avg AP_med
Baselines: GLIP T GLIP L Grounding DINO T Grounding DINO L DINOv T DINOv L GLIPv2 H MaskCLIP L FC-CLIP L OpenSeed T OpenSeed L X-Decoder T
Builds on: GLIP Grounding DINO SAM MQ-Det DINOv
Closest papers by meaning
- DoGA: Enhancing Grounded Object Detection via Grouped Pre-Training with Attributes (AAAI 2025 · similarity 0.90)
- Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation (ICML 2025 · similarity 0.90)
- Visual In-Context Prompting (CVPR 2024 · similarity 0.89)
- Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection (NeurIPS 2025 · similarity 0.89)
- Improving Open Set Recognition via Visual Prompts Distilled from Common-Sense Knowledge (AAAI 2024 · similarity 0.89)
- Training-Free Open-Ended Object Detection and Segmentation via Attention as Prompts (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.