Research map / Video / Video-language understanding
Comprehensive Visual Grounding for Video Description
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a comprehensive visual grounding network for video captioning that combines spatial-temporal entity grounding (via tracker-based label propagation) and action grounding, unified by a soft grounding supervision loss.
Problem
Grounded video captioning relies on sparse one-frame entity boxes, so attention fails to ground nouns and verbs across frames, and degraded object appearances and entity interactions hurt caption accuracy.
Approach
Propagates sparse entity boxes to adjacent frames with a tracker (detection by tracking), derives action grounding masks from the union of subject/object boxes, and supervises attention with a unified soft grounding loss over both entities and actions.
Key results
- Achieves 51.8 CIDEr on ActivityNet-Entities test, +2.3 over IAS, the best prior grounded video captioner.
- Achieves 60.2 CIDEr on MSR-VTT test, +2.2 over the best competitor (VIOLETv2 at 58.0).
- Ablation on ActivityNet-Entities val: full model reaches 54.5 CIDEr vs. 51.9 baseline; entity grounding, action grounding, and soft supervision each add gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add entity grounding (ablation) | Video captioning baseline · ActivityNet-Entities val | CIDEr: 51.9 -> 52.6 | Helped |
| Add dynamic label propagation to entity grounding (ablation) | Video captioning model · ActivityNet-Entities val | CIDEr: 52.6 -> 53.0 | Helped |
| Add action grounding (ablation) | Video captioning baseline · ActivityNet-Entities val | CIDEr: 51.9 -> 52.8 | Helped |
| Add action grounding to entity grounding and DLP (ablation) | Video captioning model · ActivityNet-Entities val | CIDEr: 51.9 -> 53.5 | Helped |
| Add soft grounding supervision (ablation) | Entity grounding + DLP + action grounding · ActivityNet-Entities val | CIDEr: 53.5 -> 54.5 | Helped |
| Vary the entity label propagation confidence threshold sth (ablation) | Entity grounding with dynamic label propagation · ActivityNet-Entities val | captioning performance | Mixed |
| Use soft grounding supervision instead of treating all generated annotations as equally important (ablation) | Grounded video captioning model · ActivityNet-Entities val | CIDEr: +2.6 | Helped |
Limitations
- MSR-VTT has no grounding annotations, so seed boxes come from GLIPv2 detections instead of human labels.
- Pseudo boxes from the tracker can be wrong; performance depends on a confidence threshold s_th.
Experimental setup
Datasets: ActivityNet-Entities MSR-VTT
Benchmarks: ActivityNet-Entities MSR-VTT
Metrics: CIDEr BLEU@1 BLEU@4 METEOR ROUGE-L SPICE
Baselines: Mask-TF BiLSTM+TA Cyclical GVD KNN-HAST IAS SwinBERT VIOLETv2 MARN OA-BTG SAAT ORG-TRL
Builds on: GVD ToMP Video Swin Transformer Text4Vis GLIPv2
Closest papers by meaning
- Grounded Video Description (CVPR 2019 · similarity 0.91)
- Exploiting Auxiliary Caption for Video Grounding (AAAI 2024 · similarity 0.90)
- VideoGEM: Training-free Action Grounding in Videos (CVPR 2025 · similarity 0.90)
- VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.89)
- Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.88)
- Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.