Research map / Detection and segmentation / Open-set and few-shot detection
OpenVIS: Open-vocabulary Video Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstFormer, an open-vocabulary video instance segmentation framework that adapts CLIP via instance-guided attention and LoRA, trained only on YouTube-VIS (40 categories), with a rollout tracker that predicts next-frame instance tokens.
Problem
Video instance segmentation restricted to training categories; the task is to detect, segment, and track arbitrary open-vocabulary object categories in videos when trained on limited categories.
Approach
Combines a COCO-pretrained Mask2Former proposal network (with a contrastive instance margin loss to diversify queries) with InstCLIP, a LoRA-adapted CLIP ViT-B/32 using instance-guidance attention masks to produce per-instance tokens in one pass, and a history-aware RNN rollout tracker trained to predict next-frame instance tracking tokens, which is category-agnostic.
Key results
- InstFormer reaches 4.2 AP on BURST (vs. 3.7 for OV2Seg) and 12.2 AP on LVVIS-novel (vs. 11.9 for OV2Seg), trained only on YouTube-VIS.
- On YouTube-VIS, InstFormer achieves 51.8 AP, outperforming MinVIS-CLIP (30.6 AP) by 21.2 AP and surpassing fully-supervised DVIS (51.2 AP).
- On UVO, InstFormer obtains 16.7 AP, above the MRCNN baselines (7.6 and 11.2 AP).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace direct masked-image CLIP classification with InstCLIP (ablation) | MinVIS-CLIP · BURST | AP: 2.1 to 3.3 | Helped |
| replace direct masked-image CLIP classification with InstCLIP (ablation) | MinVIS-CLIP · YouTube-VIS | AP: 30.6 to 48.6 | Helped |
| add contrastive instance margin loss (ablation) | InstFormer mask proposal network · UVO | AP: 13.2 to 15.8 | Helped |
| add InstCLIP instance token for association (ablation) | InstFormer · BURST | AP: 3.5 to 3.9 | Helped |
| add rollout tracker with RNN and predictive tracking loss (ablation) | InstFormer · BURST | AP: 3.9 to 4.2 | Helped |
| replace RNN rollout tracker with a linear layer (ablation) | InstFormer with instance token · BURST | AP: 4.2 to 3.7 | Hurt |
| replace RNN rollout tracker with a two-layer MLP (ablation) | InstFormer with instance token · BURST | AP: 4.2 to 3.4 | Hurt |
| add instance tokens without instance guidance (ablation) | CLIP on BURST · BURST | AP_All: 2.11 to 1.09 | Hurt |
| add binary instance-mask guidance (ablation) | InstCLIP · BURST | AP_All: 1.09 to 1.70 | Helped |
| use designed instance guidance instead of binary guidance (ablation) | InstCLIP · BURST | AP_All: 1.70 to 3.28 | Helped |
| add register token (ablation) | InstCLIP with designed guidance · BURST | AP_All: 3.28 to 3.87 | Helped |
| tune query and value projectors with LoRA instead of FFN (ablation) | CLIP in InstCLIP · BURST | AP_All: 3.5 to 3.9 | Helped |
Surprising findings
- The open-vocabulary InstFormer surpasses fully-supervised VIS methods such as DVIS and IDOL on YouTube-VIS despite the open-vocabulary design.
- Directly adding N instance tokens to CLIP without guidance performs poorly (AP 1.09 vs 2.11 for N-times CLIP on BURST).
Limitations
- Evaluation on BURST and LVVIS involves only partial category overlap; 103 categories are seen during training including COCO pretraining.
- Mask proposal network relies on a COCO-pretrained Mask2Former rather than training only on limited categories.
Experimental setup
Datasets: YouTube-VIS COCO BURST LVVIS UVO
Benchmarks: BURST LVVIS UVO YouTube-VIS
Metrics: AP AR AP50 AP75 AR10
Baselines: MinVIS MinVIS-CLIP Detic-SORT Detic-OWTB OV2Seg MaskTrack RCNN SAM-PT TAM DVIS IDOL
Builds on: Mask2Former CLIP MinVIS LoRA OV2Seg
Closest papers by meaning
- Mask-Free OVIS: Open-Vocabulary Instance Segmentation Without Manual Mask Annotations (CVPR 2023 · similarity 0.90)
- InsPro: Propagating Instance Query and Proposal for Online Video Instance Segmentation (NeurIPS 2022 · similarity 0.89)
- OW-VISCapTor: Abstractors for Open-World Video Instance Segmentation and Captioning (NeurIPS 2024 · similarity 0.88)
- Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation (CVPR 2025 · similarity 0.87)
- OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformer (ICLR 2025 · similarity 0.87)
- OVT-B: A New Large-Scale Benchmark for Open-Vocabulary Multi-Object Tracking (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.