Research map / Video / Video-language understanding
CoSTA: End-to-End Comprehensive Space-Time Entanglement for Spatio-Temporal Video Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoSTA, an end-to-end one-stage transformer for spatio-temporal video grounding that entangles space and time in video encoding, multi-modal fusion, and tube decoding via an entangled query.
Problem
Spatio-temporal video grounding suffers from insufficient space-time interaction in existing one-stage methods, causing imprecise event temporal boundaries and inconsistent object predictions across frames.
Approach
A space-time collaborative video encoder combining 2D and 3D ResNet features, a factorized spatio-temporal multi-modal Transformer, and an entangled decoder in which the temporal boundary prediction feeds into object (anchor box) queries for spatial localization.
Key results
- On HC-STVG v1, CoSTA reaches 38.97 m_vIoU and 38.19 vIoU@0.5, beating STCAT (35.09 and 30.09).
- On VidSTG interrogative sentences, CoSTA reaches 29.86 m_vIoU versus 28.22 for STCAT.
- Removing the spatio-temporal encoder drops vIoU@0.5 from 38.19 to 28.34 on HC-STVG v1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add space-time entanglement to the feature encoder (ablation) | CoSTA · HC-STVG v1 test | vIoU@0.5: 28.34 -> 38.19 | Helped |
| add an entangled query linking temporal boundaries and spatial localization (ablation) | CoSTA · HC-STVG v1 test | vIoU@0.5: 36.14 -> 38.19 | Helped |
| use both 2D and 3D backbones rather than only a 2D backbone (ablation) | CoSTA visual encoder · HC-STVG v1 test | m vIoU: 37.57 -> 38.97 | Helped |
| use both 2D and 3D backbones rather than only a 3D backbone (ablation) | CoSTA visual encoder · HC-STVG v1 test | m vIoU: 36.08 -> 38.97 | Helped |
| fuse spatial and temporal feature maps using sum with feature normalization instead of concatenation (ablation) | CoSTA video encoder · HC-STVG v1 test | m vIoU: 32.67 -> 38.97 | Helped |
| fuse spatial and temporal feature maps using sum with feature normalization instead of Hadamard product (ablation) | CoSTA video encoder · HC-STVG v1 test | m vIoU: 34.39 -> 38.97 | Helped |
| increase temporal ground-truth ratio from 0% to 90% (ablation) | CoSTA entangled-query training · HC-STVG v1 test | vIoU@0.5: 37.23 -> 38.19 | Helped |
| increase temporal ground-truth ratio from 90% to 100% (ablation) | CoSTA entangled-query training · HC-STVG v1 test | vIoU@0.5: 38.19 -> 38.10 | Hurt |
| use CoSTA rather than STCAT | HC-STVG v1 test · HC-STVG v1 test | m vIoU: 35.09 -> 38.97 | Helped |
| use CoSTA rather than STCAT | VidSTG declarative sentences · VidSTG declarative test | m vIoU: 33.14 -> 35.09 | Helped |
| use CoSTA rather than STCAT | VidSTG interrogative sentences · VidSTG interrogative test | m vIoU: 28.22 -> 29.86 | Helped |
Surprising findings
- A 3D-backbone-only encoder gives higher m_tIoU than 2D-only but lower m_vIoU and vIoU@0.5, showing spatial misalignment from temporal convolution.
Limitations
- Model has about 235M parameters, larger than prior one-stage baselines.
- Training with all ground-truth temporal boundaries causes exposure bias, so a mix of ground truth and predictions is needed.
Experimental setup
Datasets: HC-STVG VidSTG
Benchmarks: HC-STVG v1 VidSTG
Metrics: m_vIoU vIoU@0.3 vIoU@0.5 m_tIoU sIoU
Baselines: STGRN STGVT OMRN STVGBert TubeDETR STCAT
Builds on: TubeDETR STCAT DETR Deformable DETR (anchor boxes) RoBERTa
Closest papers by meaning
- What When and Where? Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions (CVPR 2024 · similarity 0.87)
- Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding (ICLR 2025 · similarity 0.86)
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.86)
- STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding (CVPR 2025 · similarity 0.86)
- Context-Guided Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.86)
- VideoGEM: Training-free Action Grounding in Videos (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.