Research map / Video / Video segmentation and tracking
Aligning Instance Brownian Bridge with Texts for Open-Vocabulary Video Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BriVIS, which links frame-level instance features as a Brownian bridge and aligns the bridge center with class texts (BTA) for open-vocabulary video instance segmentation, allowing inference on only head and tail frames.
Problem
Open-vocabulary video instance segmentation methods align each frame independently with class texts via image-text VLP models, discarding instance movement context and incurring high inference cost.
Approach
Model instance movement as a Brownian bridge between head and tail frame instance embeddings; align the bridge center (average of head and tail) with class text via a contrastive loss, plus bridge contrastive and head-tail hinge losses, so only start and end frames need inference at test time.
Key results
- BriVIS reaches 5.7 mAP on BURST and 20.9 mAP on LV-VIS (R50, cross-dataset), +2.2 and +6.7 over OV2Seg/OpenVIS-level baselines.
- With Swin-B, BriVIS reaches 7.0 mAP on BURST and 22.1 mAP on LV-VIS, improving over OV2Seg's 4.9 and 21.1.
- Using only head and tail frames for alignment cuts speed cost 32% (2.77 to 1.88 s/iter) while dropping LV-VIS mAP only 0.2 (21.1 to 20.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply Bridge-Text Alignment (BTA) | BriVIS · BURST all, cross-dataset, R50 | mAP: 3.7 -> 5.7 | Helped |
| apply Bridge-Text Alignment (BTA) | BriVIS · LV-VIS val, cross-dataset, R50 | mAP: 14.2 -> 20.9 | Helped |
| apply Bridge-Text Alignment (BTA) | BriVIS · BURST overall, base-novel, R50 | mAP: 3.5 -> 5.7 | Helped |
| apply Bridge-Text Alignment (BTA) | BriVIS · LV-VIS overall, base-novel, R50 | mAP: 12.0 -> 20.9 | Helped |
| add BTA during training (ablation) | BriVIS using all frames · BURST all | mAP: 4.4 -> 5.7 | Helped |
| add BTA during training (ablation) | BriVIS using all frames · LV-VIS val | mAP: 19.8 -> 21.1 | Helped |
| use only the first and last instance frames for alignment (ablation) | BriVIS with BTA · BURST all | mAP: 5.7 -> 5.7 | No clear effect |
| use only the first and last instance frames for alignment (ablation) | BriVIS with BTA · LV-VIS val | mAP: 21.1 -> 20.9 | Hurt |
| use only the first and last instance frames for alignment (ablation) | BriVIS with BTA · Inference speed | s/iter: 2.77 -> 1.88 | Helped |
| use only the first and last instance frames without BTA (ablation) | BriVIS · LV-VIS val | mAP: 19.8 -> 17.0 | Hurt |
Surprising findings
- Head-and-tail-only inference loses just 0.2 mAP when trained with BTA, but loses 2.8 mAP without BTA.
Limitations
- Fast-moving instance gains come with mediocre performance on slow-moving instances
- Evaluated only on OVVIS with a fixed training set of YouTube-VIS 2019 and COCO
Experimental setup
Datasets: YouTube-VIS 2019 COCO
Benchmarks: BURST LV-VIS
Metrics: mAP
Baselines: Detic-O OV2Seg OpenVIS IDOL CTVIS
Builds on: Mask2Former CLIP DVIS SideAdapter OpenVIS
Closest papers by meaning
- Minimizing Labeled, Maximizing Unlabeled: An Image-Driven Approach for Video Instance Segmentation (CVPR 2025 · similarity 0.87)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.87)
- Anchor-Aware Similarity Cohesion in Target Frames Enables Predicting Temporal Moment Boundaries in 2D (CVPR 2025 · similarity 0.86)
- OpenVIS: Open-vocabulary Video Instance Segmentation (AAAI 2025 · similarity 0.86)
- DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.86)
- VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.