Research map / Video / Video segmentation and tracking
UniVS: Unified and Universal Video Segmentation with Prompts as Queries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniVS, a single unified video segmentation model that uses prompts as queries (averaged prompt features plus target-wise prompt cross-attention) to handle category-specified and prompt-specified video segmentation tasks by converting them into prompt-guided target segmentation.
Problem
Unifying all video segmentation tasks, both category-specified (VIS, VSS, VPS) and prompt-specified (VOS, RefVOS, PVOS), in one architecture is hard because their focuses (per-frame detection and association vs. re-identifying a prompted target) conflict.
Approach
Average the prompt features of each target stored in a memory pool to form its initial decoder query, add a target-wise prompt cross-attention layer to integrate prompt memory, and feed predicted masks back as visual prompts so all VS tasks become prompt-guided target segmentation with no heuristic cross-frame matching.
Key results
- UniVS (SwinL) reaches 60.0 mAP on YT19, 57.9 on YT21, 41.7 on OVIS, 59.8 mIoU on VSPW and 92.3 mVC8 on VSPW among unified models.
- UniVS with SwinL reports 68.6 J&F on RefYT-style VOS results and 68.2/68.6 on VIPOSeg across SwinB/SwinL backbones (Gth&sf), outperforming UNINEXT ViT-H on several VOS/PVOS entries only in some settings.
- Ablation on ResNet50: prompt-as-queries plus ProCA raises OVIS mAP from 17.2 to 21.7 and YT18 J&F from 58.6 to 67.4 approximately; unified VS->P streaming inference boosts all tasks by about 6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add prompt features as decoder queries (ablation) | UniVS · YouTube-VIS 2021 development set | mAP: 45.9 -> 39.9 | Hurt |
| Add target-wise prompt cross-attention (ProCA) to prompt-query decoding (ablation) | UniVS · OVIS development set | mAP: 10.0 -> 21.7 | Helped |
| Use prompt queries with ProCA (ablation) | UniVS · VIPSeg | VPQ: 40.6 -> 35.4 | Hurt |
| Transform all video segmentation tasks into prompt-guided target segmentation for inference (ablation) | UniVS · VIPSeg | STQ: 37.9 -> 49.2 | Helped |
| Use the unified VS-to-prompt inference process instead of query-embedding similarity tracking (ablation) | UniVS · YouTube-VOS 2018 | Gth: 44.5 -> 49.2 | Helped |
| Add long-video fine-tuning stage (stage 3) (ablation) | UniVS with VS-to-prompt inference · VIPSeg | STQ: 49.2 -> 45.8 | Hurt |
Surprising findings
- Using query embedding similarity as a tracker performs poorly compared with the prompt-guided streaming inference, indicating matching-based association is weak in complex scenes.
- UniVS trails UNINEXT on instance-level VIS and prompt-specified VOS/RefVOS benchmarks, attributed to fewer object queries and less video data.
Limitations
- Performance on prompt-specified VS tasks (VOS, RefVOS, PVOS) is slightly lower than specialized or UNINEXT models.
- Limited video training data with low scene diversity restricts universality and may cause task conflicts.
- Lacks long-term information propagation modules, so it underperforms heavy offline trackers such as UniRef on long-term VOS.
Experimental setup
Datasets: YouTube-VIS 2019 YouTube-VIS 2021 OVIS VSPW VIPSeg DAVIS YouTube-VOS 2018 MOSE BURST RefDAVIS Ref-YouTube-VOS VIPOSeg
Benchmarks: YouTube-VIS 2019 YouTube-VIS 2021 OVIS VSPW VIPSeg DAVIS YouTube-VOS 2018 Ref-DAVIS Ref-YouTube-VOS VIPOSeg
Metrics: mAP mIoU mVC8 VPQ STQ J&F
Baselines: UniRef TarVIS UNINEXT Tube-Link DVIS Video-kMax VideoK-Net TubeFormer GenVIS XMem DeAOT SgMg
Builds on: Mask2Former SEEM CLIP DVIS UNINEXT
Closest papers by meaning
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.87)
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.86)
- Prompt-Driven Referring Image Segmentation with Instance Contrasting (CVPR 2024 · similarity 0.86)
- Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.86)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.86)
- Unsupervised Universal Image Segmentation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.