Research map / Video / Video-language understanding
Context-Guided Spatio-Temporal Video Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CG-STVG, a DETR-style spatio-temporal video grounding model that mines instance appearance and motion context from the video at each decoding stage and refines it with temporal and spatial confidence filtering, reaching new SOTA on HCSTVG-v1/v2 and VidSTG.
Problem
Spatio-temporal video grounding degrades under distractors and large object appearance variations because the text query alone gives insufficient object information.
Approach
Instead of enriching the text query, mine instance visual context (appearance and motion RoI features from predicted foreground regions) inside the video at each decoding stage, refine it with temporal and spatial confidence filters, and feed it to the next stage as guidance for target localization.
Key results
- On HCSTVG-v1 test, CG-STVG reaches 52.8 m_tIoU and 38.4 m_vIoU, improving over the STCAT m_tIoU of 49.4 by 3.4 points and the CSDVL m_vIoU of 36.9 by 1.5 points.
- On HCSTVG-v2 val, CG-STVG reaches 60.0 m_tIoU and 39.5 m_vIoU, outperforming CSDVL (58.1, 38.7).
- On VidSTG test, CG-STVG improves over its baseline by 1.7 m_tIoU and 1.6 m_vIoU on declarative sentences and by 1.1 and 1.3 on interrogative sentences.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ICG and ICR instance context guidance (ablation) | CG-STVG · HCSTVG-v1 test | m vIoU: 36.52 -> 38.42 | Helped |
| Add ICG instance context generation (ablation) | CG-STVG · HCSTVG-v1 test | m vIoU: 36.52 -> 37.42 | Helped |
| Add temporal refinement to ICG (ablation) | CG-STVG · HCSTVG-v1 test | m vIoU: 37.42 -> 37.86 | Helped |
| Add spatial refinement to ICG (ablation) | CG-STVG · HCSTVG-v1 test | m vIoU: 37.42 -> 38.04 | Helped |
| Add both temporal and spatial refinement (ablation) | CG-STVG · HCSTVG-v1 test | m vIoU: 37.42 vs 38.42 | Helped |
| Set temporal threshold θt to 0.7 (ablation) | CG-STVG ICR · HCSTVG-v1 test | m tIoU: 52.82, 52.80, 52.84, 52.84 | No clear effect |
| Set spatial threshold θs to 0.8 (ablation) | CG-STVG ICR · HCSTVG-v1 test | m vIoU: 37.47, 37.44, 38.42, 37.61 | Helped |
| Use two-level confidence refinement rather than one-level addition (ablation) | CG-STVG ICR · HCSTVG-v1 test | m vIoU: 38.31 -> 38.42 | Helped |
| Use two-level confidence refinement rather than one-level multiplication (ablation) | CG-STVG ICR · HCSTVG-v1 test | m vIoU: 38.25 -> 38.42 | Helped |
| Apply instance context to TDB (ablation) | CG-STVG · HCSTVG-v1 test | m vIoU: 38.42 -> 38.01 | Hurt |
| Add CG-STVG instance context to the baseline | CG-STVG · HCSTVG-v1 test | m tIoU: 50.4 -> 52.8 | Helped |
Surprising findings
- Applying instance context to the temporal decoding block slightly hurts performance (m_vIoU 38.42 to 38.01), so context is used only in the spatial branch.
Limitations
- Instance context depends on the accuracy of estimated foreground regions, so noisy regions can introduce harmful features that need thresholded refinement.
- Thresholds for temporal and spatial refinement (0.7 and 0.8) are hand-tuned on the test-side ablations.
Experimental setup
Datasets: HCSTVG-v1 HCSTVG-v2 VidSTG
Benchmarks: HCSTVG-v1 HCSTVG-v2 VidSTG
Metrics: m_tIoU m_vIoU vIoU@0.3 vIoU@0.5
Baselines: STGVT STVGBert TubeDETR STCAT CSDVL PCC 2D-Tan MMN STGRN OMRN
Builds on: DETR MDETR TubeDETR STCAT CSDVL
Closest papers by meaning
- Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Grounding (ICLR 2025 · similarity 0.92)
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.90)
- Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding (ICLR 2025 · similarity 0.89)
- STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding (CVPR 2025 · similarity 0.89)
- VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.89)
- VideoGEM: Training-free Action Grounding in Videos (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.