Research map / Video / Video-language understanding
Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts Grounding DINO with tubelet-based spatial and temporal grounding, contextual referral queries, and a self-paced curriculum to achieve weakly supervised spatio-temporal video grounding without box labels.
Problem
Weakly supervised spatio-temporal video grounding (WSTVG): localizing the subject tubelet and its activity boundaries from free-form text using only video-level descriptions, without bounding box or temporal supervision.
Approach
Extends frozen Grounding DINO detections into tracked tubelets, adds a spatial tubelet-phrase contrastive module and a reconstruction-based temporal module, uses LLM-extracted referral sub-queries (attributes, actions) for contextual grounding, and trains with a self-paced curriculum that increases the max number of tubelets per video across stages.
Key results
- On HCSTVG-v1, CoSPaL reaches 22.1 m_vIoU versus 14.2 for the prior weakly supervised SOTA WINNER (+7.9 claimed in abstract).
- On VidSTG declarative, CoSPaL reaches 16.0 m_vIoU vs 11.6 for WINNER (+4.4 over prior weak SOTA; abstract states 3.9%).
- Ablation on HCSTVG-v1: TPG, CRG and SPS together raise m_vIoU from 9.0 (W-GDINO) to 22.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace W-GDINO with CoSPaL | Grounding DINO for weakly supervised video grounding · HCSTVG-v1 | m_vIoU: 9.0 -> 22.1 | Helped |
| Replace W-GDINO with CoSPaL | Grounding DINO for weakly supervised video grounding · HCSTVG-v2 | m_vIoU: 9.9 -> 22.2 | Helped |
| Replace W-GDINO with CoSPaL | Grounding DINO for weakly supervised video grounding · VidSTG-Declarative | m_vIoU: 10.6 -> 16.0 | Helped |
| Replace W-GDINO with CoSPaL | Grounding DINO for weakly supervised video grounding · VidSTG-Interrogative | m_vIoU: 9.8 -> 13.5 | Helped |
| Add temporal grounding to spatial grounding (ablation) | TPG · HCSTVG-v1 | m_vIoU: 13.5 -> 18.0 | Helped |
| Add temporal self-attention (ablation) | Spatial grounding module · HCSTVG-v1 | m_vIoU: 13.5 -> 13.9 | Helped |
| Add SPS stage II after stage I (ablation) | TPG · HCSTVG-v1 | m_vIoU: 17.7 -> 18.5 | Helped |
| Add SPS stage III after stage II (ablation) | TPG · HCSTVG-v1 | m_vIoU: 18.5 -> 20.1 | Helped |
| Add CRG to TPG (ablation) | TPG · HCSTVG-v1 | m_vIoU: 19.2 -> 21.0 | Helped |
| Add SPS to TPG and CRG (ablation) | TPG + CRG · HCSTVG-v1 | m_vIoU: 21.0 -> 22.1 | Helped |
| Replace BERT with CLIP text encoder (ablation) | TPG · HCSTVG-v1 | m_vIoU: 19.2 -> 18.8 | Hurt |
| Replace DETR with Faster-RCNN detector (ablation) | CoSPaL · HCSTVG-v1 | m_vIoU: 22.1 -> 16.4 | Hurt |
Surprising findings
- Feeding the whole caption to Grounding DINO at training and test time improves m_vIoU to 22.7 on HCSTVG-v1 over the noun-only setting (19.2), though it departs from the traditional weakly supervised protocol.
- Simply applying frozen Grounding DINO with tracking (W-GDINO) is already competitive with prior weakly supervised methods on tIoU.
Limitations
- Pipeline depends on detection and tracking quality; HCSTVG datasets have zoom shots, scene changes and identity switches where detectors and trackers fail.
- Performance remains far below fully supervised methods (e.g., VGDINO 38.3 m_vIoU on HCSTVG-v1 vs 22.1).
- Uses GPT-3.5 for query decomposition, which sometimes hallucinates which actor performs an action.
Experimental setup
Datasets: VidSTG HCSTVG-v1 HCSTVG-v2
Benchmarks: VidSTG HCSTVG-v1 HCSTVG-v2
Metrics: m_vIoU vIoU@0.3 vIoU@0.5 tIoU
Baselines: W-GDINO WINNER AWGU Vis-CTX TubeDETR STCAT CSDVL CG-STVG VGDINO
Builds on: Grounding DINO self-paced learning BoTSORT
Closest papers by meaning
- STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding (CVPR 2025 · similarity 0.92)
- What When and Where? Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions (CVPR 2024 · similarity 0.90)
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.90)
- Context-Guided Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.89)
- VideoGEM: Training-free Action Grounding in Videos (CVPR 2025 · similarity 0.88)
- VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.