Research map / Detection and segmentation / Open-set and few-shot detection
Visual In-Context Prompting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DINOv, a unified visual in-context prompting framework for generic (open-set) and referring segmentation, trained jointly on COCO and SA-1B on a MaskDINO-based encoder-decoder.
Problem
Visual prompting methods mainly support referring segmentation of a single object and do not address generic tasks like open-set segmentation and detection that require segmenting multiple objects of a concept from in-context visual examples.
Approach
A query-based encoder-decoder with a prompt encoder that extracts reference visual prompt features from reference image-prompt pairs via masked cross-attention, a shared decoder that attends them to the target image, and separate generic (learnable) and interactive point queries, enabling both generic and referring segmentation with arbitrary numbers of in-context examples.
Key results
- DINOv-L reaches 57.7 PQ on COCO (in-domain) versus 58.3 PQ for its MaskDINO-L baseline, and surpasses SegGPT-L by 14.3 PQ.
- DINOv-L attains 23.2 PQ on ADE20K zero-shot, above FC-CLIP-L (26.8 PQ is FC-CLIP; DINOv exceeds the text-prompted OpenSeed-L at 19.7 PQ).
- DINOv-T achieves 73.3 J&F on DAVIS17 zero-shot video object segmentation, outperforming SEEM-L (58.9) without video training data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use visual in-context prompting with DINOv-L rather than MaskDINO-L | COCO segmentation · COCO | PQ: 58.3 -> 57.7 | Hurt |
| use DINOv-L visual prompts rather than the OpenSeed-L baseline | generic segmentation · SegInW | AP-Average: 36.1 -> 40.6 | Helped |
| use DINOv-L visual prompts rather than the OpenSeed-L baseline | object detection · ODinW | Average AP: 15.2 -> 15.7 | Helped |
| use only point queries instead of both generic and point queries (ablation) | DINOv-SwinT · COCO | PQ: 49.6 -> 45.2 | Hurt |
| use only generic queries instead of both generic and point queries (ablation) | DINOv-SwinT · COCO | PQ: 49.6 -> 46.2 | Hurt |
| encode visual prompts with cropped-region CLIP features rather than the proposed prompt encoder (ablation) | DINOv-SwinT · ADE20K | mask AP: 11.4 -> 1.4 | Hurt |
| train on both COCO and SAM rather than COCO only (ablation) | DINOv-SwinT · COCO | PQ: 48.9 -> 49.6 | Helped |
| train on both COCO and SAM rather than COCO only (ablation) | DINOv-SwinT · DAVIS17 | JF: 63.3 -> 73.3 | Helped |
| increase prompt-sampling batch size from 1 to 64 (ablation) | DINOv-SwinT · COCO | PQ: 28.9 -> 49.0 | Helped |
| increase the number of in-context examples (ablation) | DINOv · generic segmentation | performance | Mixed |
Surprising findings
- Replacing the proposed prompt encoder with a pre-trained CLIP encoder on cropped prompt regions markedly hurts open-set ADE20K generalization (mIoU 13.3 vs 21.9 for Swin-T).
- Prompting FC-CLIP with CLIP visual features performs far worse than its text-prompted results, suggesting text-based open-set models do not transfer easily to visual prompts.
Limitations
- Uses only limited semantically labeled data (COCO), which could be scaled up.
- Not extended to text prompts for multi-modal understanding yet.
Experimental setup
Datasets: COCO SA-1B
Benchmarks: COCO ADE20K SegInW ODinW DAVIS17 DAVIS16-Interactive YouTube-VOS 2018
Metrics: PQ mask AP box AP mIoU AP-Average AP-Median JF J F
Baselines: Mask2Former MaskDINO OneFormer X-Decoder OpenSeed FC-CLIP ODISE SegGPT Painter SEEM PerSAM UNINEXT
Builds on: MaskDINO Semantic-SAM SEEM SegGPT
Closest papers by meaning
- Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation (ICML 2025 · similarity 0.89)
- Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm (AAAI 2025 · similarity 0.89)
- Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach (CVPR 2024 · similarity 0.89)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.89)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.88)
- Exploring Task-Level Optimal Prompts for Visual In-Context Learning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.