Research map / Video / Video-language understanding
Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TA-STVG, a Transformer for spatio-temporal video grounding that initializes object queries from target-aware cues (text-guided temporal sampling plus attribute-aware spatial activation) instead of zeros, reaching SOTA on HCSTVG-v1/v2 and VidSTG.
Problem
Zero-initialized object queries in Transformer-based spatio-temporal video grounding lack target-specific cues, so they struggle to learn discriminative target localization in scenes with distractors or occlusion.
Approach
Generate spatial and temporal object queries directly from the video-text pair: TTS selects target-relevant frames guided by holistic text features, then ASA uses attribute-supervised cross-attention on the subject to activate fine-grained spatial features that initialize the queries.
Key results
- On HCSTVG-v1 test, TA-STVG reaches 53.0 m_tIoU, 39.1 m_vIoU, 63.1 vIoU@0.3 and 36.8 vIoU@0.5, versus 49.9/36.4/57.6/32.1 for the zero-query baseline.
- On VidSTG, TA-STVG reports the best results on all 8 metrics, e.g. 51.7 m_tIoU and 34.4 m_vIoU on declarative sentences, above CG-STVG's 51.4 and 34.0.
- Plugging TTS and ASA into TubeDETR and STCAT (retrained) improves m_tIoU by 2.3 and 1.7 points respectively on HCSTVG-v1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace zero-initialized queries with TTS+ASA target-aware queries | TA-STVG · HCSTVG-v1 test | m_tIoU: 49.9 -> 53.0 | Helped |
| add text-guided temporal sampling (TTS) without ASA (ablation) | TA-STVG · HCSTVG-v1 test | m_tIoU: 49.9 -> 52.2 | Helped |
| add attribute-aware spatial activation (ASA) without TTS (ablation) | TA-STVG · HCSTVG-v1 test | m_tIoU: 49.9 -> 51.4 | Helped |
| add TTS and ASA together (ablation) | TA-STVG · HCSTVG-v1 test | m_tIoU: 49.9 -> 53.0 | Helped |
| add text-guided appearance branch (ablation) | TA-STVG with ASA · HCSTVG-v1 test | m_tIoU: 51.4 -> 51.8 | Helped |
| add text-guided motion branch (ablation) | TA-STVG with ASA · HCSTVG-v1 test | m_tIoU: 51.4 -> 52.3 | Helped |
| use both appearance and motion branches (ablation) | TA-STVG with ASA · HCSTVG-v1 test | m_tIoU: 51.4 -> 53.0 | Helped |
| remove textual guidance from TTS (ablation) | TA-STVG with appearance and motion branches · HCSTVG-v1 test | m_tIoU: 53.0 -> 51.8 | Hurt |
| add appearance attribute (ablation) | TA-STVG with TTS · HCSTVG-v1 test | m_tIoU: 52.2 -> 52.3 | Helped |
| add motion attribute (ablation) | TA-STVG with TTS · HCSTVG-v1 test | m_tIoU: 52.2 -> 52.7 | Helped |
| use both appearance and motion attributes (ablation) | TA-STVG with TTS · HCSTVG-v1 test | m_tIoU: 52.2 -> 53.0 | Helped |
| add subject guidance to ASA (ablation) | TA-STVG with appearance and motion attributes · HCSTVG-v1 test | m_tIoU: 52.6 -> 53.0 | Helped |
Surprising findings
- Oracle experiment: initializing queries from ground-truth boxes raises HCSTVG-v1 m_tIoU from 49.9 to 68.9, showing large headroom from target-aware queries.
Limitations
- High computational demand, since all sampled frames must be processed at once.
- If TTS selects no target-containing frames, the generated queries lack target information and may degrade performance.
- Attribute labels rely on Stanford CoreNLP parsing of the subject and attributes.
Experimental setup
Datasets: HCSTVG-v1 HCSTVG-v2 VidSTG
Benchmarks: HCSTVG-v1 HCSTVG-v2 VidSTG
Metrics: m_tIoU m_vIoU vIoU@0.3 vIoU@0.5
Baselines: STVGBert TubeDETR STCAT SGFDN STVGFormer CG-STVG PCC 2D-Tan MMN STGRN OMRN STGVT
Builds on: DETR TubeDETR STCAT MDETR VidSwin
Closest papers by meaning
- Context-Guided Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.92)
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.90)
- STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding (CVPR 2025 · similarity 0.89)
- Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding (ICLR 2025 · similarity 0.87)
- VideoGEM: Training-free Action Grounding in Videos (CVPR 2025 · similarity 0.87)
- VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.