Research map / Detection and segmentation / Open-set and few-shot detection
OW-VISCapTor: Abstractors for Open-World Video Instance Segmentation and Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OW-VISCapTor, which uses an object abstractor with SAM-style point-grid prompt encoding plus an object-to-text abstractor with masked cross-attention feeding a frozen OPT-2.7B, to jointly detect, segment, track, and caption open-world and closed-world objects in videos.
Problem
Open-world video instance segmentation and object-centric captioning have been handled separately; existing abstractor-based open-world object discovery loses fine spatial detail, and captioning is limited to closed-world labels.
Approach
Spatially-rich open-world object queries from a SAM prompt encoder over an evenly spaced point grid, an inter-query L1 contrastive loss for diversity, and an object-to-text abstractor whose cross-attention is masked to each object's predicted mask before feeding a frozen LLM for captions.
Key results
- On BURST validation, OW-VISCapTor+DEVA reaches 55.2 OWTA on unseen categories versus 49.6 for the best specialized baseline EntitySeg+DEVA (~6 points), and 50.0 unseen with CAROQ versus 42.3 for the generalized baseline.
- On VidSTG Dense VOC, OW-VISCapTor+CAROQ attains 43.9 CapA and 53.1 CHOTA, versus 36.8 CapA and 51.6 CHOTA for DVOC-DS joint training.
- Ablation on BURST: removing the prompt encoder drops unseen OWTA from 43.8 to 41.2; removing the contrastive loss drops it to 41.6; removing masked attention drops VidSTG CapA from 43.9 to 20.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use OW-VISCapTor+DEVA instead of EntitySeg+DEVA | BURST validation · BURST validation | OWTA (Unseen): 49.6 -> 55.2 | Helped |
| use OW-VISCapTor+DEVA instead of Mask2Former+DEVA | BURST validation · BURST validation | OWTA (Overall): 69.5 -> 69.0 | Hurt |
| use OW-VISCapTor+CAROQ instead of DVOC-DS joint training | VidSTG · VidSTG | CapA: 36.8 -> 43.9 | Helped |
| use OW-VISCapTor+CAROQ instead of DVOC-DS joint training | VidSTG · VidSTG | CHOTA: 51.6 -> 53.1 | Helped |
| add the prompt encoder to form spatially rich open-world queries (ablation) | OW-VISCapTor · BURST validation | OWTA (Unseen): 41.2 -> 43.8 | Helped |
| add inter-query contrastive loss (ablation) | OW-VISCapTor · BURST validation | OWTA (Unseen): 41.6 -> 43.8 | Helped |
| add inter-query contrastive loss (ablation) | OW-VISCapTor · OVIS validation | AP: 23.2 -> 25.4 | Helped |
| add masked attention in the object-to-text abstractor (ablation) | OW-VISCapTor · VidSTG | CapA: 20.3 -> 43.9 | Helped |
| caption object crops rather than using masked attention (ablation) | OW-VISCapTor · VidSTG | CapA: 43.9 -> 36.6 | Hurt |
| enlarge bounding boxes by 10% before cropping for captions (ablation) | bounding-box captioning · VidSTG | CapA: 36.6 -> 39.4 | Helped |
| use clip length T=2 instead of T=1 (ablation) | OW-VISCapTor with CAROQ · VidSTG | CHOTA: 49.2 -> 53.0 | Helped |
| use clip length T=4 instead of T=2 (ablation) | OW-VISCapTor with CAROQ · VidSTG | CHOTA: 53.0 -> 52.3 | Hurt |
Surprising findings
- Dropping masked attention in the object-to-text abstractor reduces captioning accuracy by 23 points, showing that concatenating object queries with text embeddings alone does not give object-centric captions.
Limitations
- Fails to detect some open-world objects of interest and sometimes produces meaningless captions for small objects.
- Object identities are lost after prolonged occlusion (~30 frames).
- Underperforms DVOC-DS on DetA and AssA in Dense VOC, and ranks second on common categories in BURST.
Experimental setup
Datasets: BURST VidSTG OVIS COCO
Benchmarks: BURST VidSTG OVIS
Metrics: OWTA CHOTA DetA AssA CapA AP_M AP AP50 AR1 AR10
Baselines: OWTB Mask2Former+STCN Mask2Former+DEVA EntitySeg+DEVA DVOC-DS Mask2Former+DEVA+BLIP2 CAROQ MaskTrack DeVIS MinVIS VITA InstanceFormer
Builds on: Mask2Former SAM BLIP-2 DEVA CAROQ DVOC-DS
Closest papers by meaning
- Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.88)
- OpenVIS: Open-vocabulary Video Instance Segmentation (AAAI 2025 · similarity 0.88)
- Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos (NeurIPS 2025 · similarity 0.88)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.87)
- v-CLR: View-Consistent Learning for Open-World Instance Segmentation (CVPR 2025 · similarity 0.87)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.