Research map / Video / Video-language understanding
VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoGrounding-DINO, a DETR-style spatio-temporal video grounding model built on frozen-encoder Grounding DINO with temporal aggregation, evaluated in closed-set and zero-shot open-vocabulary settings.
Problem
Spatio-temporal video grounding models trained on small closed-set datasets (VidSTG, HC-STVG) generalize poorly to unseen vocabularies and distributions.
Approach
Initialize the spatial modules of a DETR-like video grounding model from the pretrained Grounding DINO image grounding model, keep the vision and text encoders frozen, and add temporal self-attention and learnable adapter/aggregation blocks plus a temporal start/end head to ground text queries in video.
Key results
- Closed-set VidSTG declarative m_vIoU 34.67 vs STCAT 33.14; HC-STVG V1 m_vIoU 38.25 vs STVGFormer 36.90; HC-STVG V2 m_vIoU 39.88 vs STVGFormer 38.70.
- Open-vocabulary (trained on VidSTG, tested on HC-STVG V1) m_vIoU 27.46 vs STCAT 22.58 and TubeDETR 16.84.
- YouCook-Interactions accuracy 57.73 vs STCAT 55.90 and TubeDETR 51.63.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VideoGrounding-DINO instead of STVGFormer | VidSTG test, declarative · VidSTG test, declarative | m vIoU: 33.70 -> 34.67 | Helped |
| use VideoGrounding-DINO instead of STVGFormer | HC-STVG V1 test, closed-set · HC-STVG V1 test, closed-set | m vIoU: 36.90 -> 38.25 | Helped |
| use VideoGrounding-DINO instead of STVGFormer | HC-STVG V2 val, closed-set · HC-STVG V2 val, closed-set | m vIoU: 38.70 -> 39.88 | Helped |
| use VideoGrounding-DINO instead of STCAT | HC-STVG V1, open-vocabulary · HC-STVG V1, open-vocabulary | m vIoU: 22.58 -> 27.46 | Helped |
| use VideoGrounding-DINO instead of STCAT | YouCook-Interactions, open-vocabulary · YouCook-Interactions, open-vocabulary | accuracy: 55.90 -> 57.73 | Helped |
| add decoder temporal aggregation (ablation) | VideoGrounding-DINO · VidSTG test, declarative | m vIoU: 18.07 -> 20.74 | Helped |
| add encoder temporal aggregation (ablation) | VideoGrounding-DINO · VidSTG test, declarative | m vIoU: 20.74 -> 23.19 | Helped |
| fine-tune spatial modules in the decoder (ablation) | VideoGrounding-DINO · VidSTG test, declarative | m vIoU: 23.19 -> 28.97 | Helped |
| fine-tune spatial modules in the encoder (ablation) | VideoGrounding-DINO · VidSTG test, declarative | m vIoU: 28.97 -> 34.67 | Helped |
Surprising findings
- Naive frozen Grounding DINO with no temporal modules performs weakly on VidSTG (declarative m_vIoU 18.07), showing temporal modeling is essential.
Limitations
- Relies on image-text pretrained grounding models rather than video-language pretraining, limiting open-vocabulary gains.
- Open-vocabulary evaluation covers only two datasets with limited distribution shift analysis.
- Video-language pretraining dataset for open-vocabulary grounding is not available.
Experimental setup
Datasets: VidSTG HC-STVG V1 HC-STVG V2 YouCook-Interactions
Benchmarks: VidSTG HC-STVG V1 HC-STVG V2 YouCook-Interactions
Metrics: m_tIoU m_vIoU vIoU@0.3 vIoU@0.5 accuracy (pointing game)
Baselines: TubeDETR STCAT STVGFormer OMRN STVGBert Augmented 2D-TAN STGRN Frozen Grounding DINO
Builds on: Grounding DINO DINO GLIP TubeDETR Deformable DETR
Closest papers by meaning
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.90)
- VideoGEM: Training-free Action Grounding in Videos (CVPR 2025 · similarity 0.90)
- Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding (NeurIPS 2025 · similarity 0.89)
- Context-Guided Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.89)
- Comprehensive Visual Grounding for Video Description (AAAI 2024 · similarity 0.89)
- What When and Where? Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.