Research map / Video / Video-language understanding
STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts Grounding DINO to weakly supervised spatio-temporal video grounding via tubelet referral grounding (TRG) and adds curriculum learning over sub-actions (SA-TCL) and tubelet congestion (CG-SCL), reaching SOTA on VidSTG and HCSTVG.
Problem
Weakly supervised spatio-temporal video grounding (WSTVG) must localize subjects in space and time from text queries alone, without bounding box or frame-level supervision; naive vision-foundation-model adaptations lack compositional action understanding and handle dense scenes poorly.
Approach
Connects textual queries to tubelets (TRG) built from Grounding DINO detections and tracks, then applies two curricula: sub-action temporal curriculum over GPT-3.5-decomposed ordered action phrases, and congestion-guided spatial curriculum ordering samples by average pairwise tubelet tIoU, plus soft-label filtering.
Key results
- On HCSTVG-v1, STPro reaches 17.56 m_vIoU and 26.98 vIoU@0.3, surpassing the prior SOTA VCMA (14.64 m_vIoU) by 2.92 points.
- On VidSTG declarative, STPro achieves 15.52 m_vIoU, improving over VCMA (14.45) by about 1.07 points.
- On HCSTVG-v2 validation, STPro reaches 19.99 m_vIoU versus 9.85 for the W-GDINO baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SA-TCL to TRG (ablation) | TRG on HCSTVG-v1 · HCSTVG-v1 | m_vIoU: 15.31 -> 17.15 | Helped |
| Add CG-SCL to TRG (ablation) | TRG on HCSTVG-v1 · HCSTVG-v1 | m_vIoU: 14.81 -> 15.31 | Helped |
| Add CG-SCL and SA-TCL to TRG (ablation) | TRG on HCSTVG-v1 · HCSTVG-v1 | m_vIoU: 14.81 -> 17.56 | Helped |
| Add POS tags instead of full captions (ablation) | TRG on HCSTVG-v1 · HCSTVG-v1 | m_vIoU: 9.99 -> 10.40 | Helped |
| Add TRM to SRM (ablation) | TRG on HCSTVG-v1 · HCSTVG-v1 | m_vIoU: 9.99 -> 10.40 | Helped |
| Use sub-action temporal curriculum (ablation) | TRM on HCSTVG-v1 · HCSTVG-v1 | tIoU: 32.96 -> 33.92 | Helped |
| Use SLF (ablation) | SRM + POS on HCSTVG-v1 · HCSTVG-v1 | m_vIoU: 10.40 -> 11.92 | Helped |
| Use CGS (ablation) | SRM + POS on HCSTVG-v1 · HCSTVG-v1 | m_vIoU: 10.40 -> 11.01 | Helped |
| Use tubelets instead of frames (ablation) | STPro on HCSTVG-v1 · HCSTVG-v1 | m_vIoU: 16.64 -> 17.56 | Helped |
| Use STPro instead of VCMA | Weakly supervised grounding on VidSTG declarative · VidSTG declarative | m_vIoU: 14.45 -> 15.52 | Helped |
| Use STPro instead of VCMA | Weakly supervised grounding on VidSTG interrogative · VidSTG interrogative | m_vIoU: 13.25 -> 12.56 | Hurt |
Surprising findings
- Simply adapting Grounding DINO to videos (W-GDINO) with tracking gives only 9.04 m_vIoU on HCSTVG-v1, and the TRM temporal module frequently shifts boundaries in the wrong direction for trimmed queries (65% of start-trimmed cases).
Limitations
- Performance on VidSTG interrogative subsets is not state-of-the-art and gains are under 2% on recent VidSTG results.
- Relies on tracking (BoT-SORT) and per-frame Grounding DINO detections, which are noisy under partial bodies, poses, and cut-scenes.
- Depends on an LLM (GPT-3.5) to decompose queries into sub-actions and extract POS tags.
Experimental setup
Datasets: VidSTG HCSTVG-v1 HCSTVG-v2
Benchmarks: VidSTG HCSTVG-v1 HCSTVG-v2
Metrics: m_vIoU vIoU@0.3 vIoU@0.5 m_tIoU
Baselines: W-GDINO VCMA WINNER Vis-CTX AWGU WSSTG RAIR+CPL MATN+LCNet
Builds on: Grounding DINO CNM CPL BoT-SORT
Closest papers by meaning
- Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding (ICLR 2025 · similarity 0.92)
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.92)
- What When and Where? Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions (CVPR 2024 · similarity 0.90)
- Context-Guided Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.89)
- Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Grounding (ICLR 2025 · similarity 0.89)
- VideoGEM: Training-free Action Grounding in Videos (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.