Research map / Video / Video-language understanding
ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ViCaS, a human-annotated video dataset with detailed captions and phrase-grounded pixel masks, a two-task benchmark (captioning and language-guided video instance segmentation), a validated caption-scoring protocol, and the Video-LLaVA-Seg baseline.
Problem
Holistic video understanding (captioning, QA) and pixel-level language-guided video segmentation have evolved separately with distinct benchmarks and architectures, with no dataset that combines detailed captions with grounded object masks.
Approach
Annotate videos with detailed human captions whose object phrases are grounded to temporally consistent masks, pair this with a captioning task and a new LG-VIS task, validate open-ended caption scoring via a user study, and train an end-to-end MLLM with a SAM2-style mask decoder driven by <SEG> tokens.
Key results
- Video-LLaVA-Seg achieves 20.5 mAP and 3.0 Caption Accuracy on the ViCaS validation set, versus 13.3 mAP for DsHmp and 10.7 mAP for VideoLISA.
- Llama3-70B judge reaches Pearson r=59.3% with human scores (65.6% with average pooling over paraphrased references), close to GPT-4 at 58.9% (67.3% pooled).
- Training solely on LG-VIS yields 18.5 mAP versus 20.5 for the multi-task model, indicating captioning supervision helps segmentation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Ts=1 instead of Ts=8 slow frames (ablation) | Video-LLaVA-Seg · ViCaS validation | CA: 3.0 -> 2.1 | Hurt |
| Use Ts=1 instead of Ts=8 slow frames (ablation) | Video-LLaVA-Seg · ViCaS validation | mAP: 20.5 -> 15.9 | Hurt |
| Use Ts=2 instead of Ts=8 slow frames (ablation) | Video-LLaVA-Seg · ViCaS validation | CA: 3.0 -> 2.4 | Hurt |
| Use Ts=2 instead of Ts=8 slow frames (ablation) | Video-LLaVA-Seg · ViCaS validation | mAP: 20.5 -> 17.0 | Hurt |
| Use Ts=4 instead of Ts=8 slow frames (ablation) | Video-LLaVA-Seg · ViCaS validation | CA: 3.0 -> 2.8 | Hurt |
| Use Ts=4 instead of Ts=8 slow frames (ablation) | Video-LLaVA-Seg · ViCaS validation | mAP: 20.5 -> 18.8 | Hurt |
| Train only for LG-VIS and omit captioning training (ablation) | Video-LLaVA-Seg · ViCaS validation | mAP: 20.5 -> 18.5 | Hurt |
| Train only for captioning and omit LG-VIS training (ablation) | Video-LLaVA-Seg · ViCaS validation | CA: 3.0 -> 3.0 | No clear effect |
| Replace AM-RADIO with CLIP and remove Slow-Fast Pooling (ablation) | Video-LLaVA-Seg · ViCaS validation | mAP: 20.5 -> 19.9 | Hurt |
| Replace AM-RADIO with CLIP and remove Slow-Fast Pooling (ablation) | Video-LLaVA-Seg · ViCaS validation | CA: 3.0 -> 2.9 | Hurt |
| Use Video-LLaVA-Seg instead of LMPM | LG-VIS methods · ViCaS validation | mAP: 8.4 -> 20.5 | Helped |
| Use Video-LLaVA-Seg instead of DsHmp | LG-VIS methods · ViCaS validation | mAP: 13.3 -> 20.5 | Helped |
Surprising findings
- Word-matching metrics such as CIDEr and BLEU4 gave negative robustness margins against hard-negative captions, while LLM-based scorers were most robust.
- Captioning accuracy of the trained model (3.0) matches off-the-shelf LLaVA-OneVision (2.9) and MiniCPM-o 2.6 (3.0) despite far less training data.
Limitations
- Caption evaluation relies on an LLM judge (Llama3-70B) whose correlation with humans is about 59%, which is modest.
- Only 131 captions were used in the user study, with 50 hard-positive/negative samples.
- Annotated masks are drawn at 1 fps and propagated with SAM2, so dense masks depend on propagation quality.
Experimental setup
Datasets: ViCaS WebVid10M Panda70M Oops UVO Kinetics-700
Benchmarks: ViCaS validation set
Metrics: Caption Accuracy (CA) mAP AP50 AP75 AP90 Pearson correlation
Baselines: LLaVA-OneVision MiniCPM-o 2.6 LMPM DsHmp VideoLISA
Builds on: LLaVA SAM2 VideoLISA AM-RADIO
Closest papers by meaning
- One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos (NeurIPS 2024 · similarity 0.89)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.88)
- VideoGLaMM : A Large Multimodal Model for Pixel-Level Visual Grounding in Videos (CVPR 2025 · similarity 0.88)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.88)
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.88)
- HourVideo: 1-Hour Video-Language Understanding (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.