Research map / Video / Video-language understanding
Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free-style zero-shot spatio-temporal video grounding framework that exploits MLLM "grounding tokens" with decomposed attribute/action sub-queries, logit-guided re-attention test-time prompt tuning, and temporal-augmented assembling.
Problem
Zero-shot spatio-temporal video grounding (STVG): localizing the spatio-temporal tube of a video described by a text query without fully supervised or dense annotations, where MLLMs often ignore attribute and action cues in complex queries.
Approach
Identifies special tokens after the instruction in MLLMs as "grounding tokens" that attend to target regions; decomposes queries into attribute (spatial) and action (temporal) yes/no sub-queries and optimizes learnable visual prompts via logit-guided re-attention at test time, plus a temporal-reversal assembling step to improve spatial consistency.
Key results
- On HCSTVG-v1 with LLaVA-OneVision-7B, the method reaches 24.8 m_vIoU, 41.5 vIoU@0.3 and 16.3 vIoU@0.5, versus E3M's 19.1 m_vIoU and 10.6 vIoU@0.5.
- On HCSTVG-v2, the method outperforms the weakly-supervised CoSPaL by 5.5% on m_vIoU (LLaVA-OneVision-7B reaches 27.7 m_vIoU).
- Based on LLaVA-Next-Video-7B, the method improves over E3M by 4.2% on vIoU@0.3 and 1.8% on vIoU@0.5 on HCSTVG-v1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the full zero-shot framework instead of E3M | LLaVA-OneVision-7B · HCSTVG-v1 | m_vIoU: 19.1 -> 24.8 | Helped |
| use the full zero-shot framework instead of CoSPaL | LLaVA-OneVision-7B · HCSTVG-v2 | m_vIoU: 22.2 -> 27.7 | Helped |
| use the full zero-shot framework instead of E3M | LLaVA-OneVision-7B · VidSTG (Declarative) | m_vIoU: 16.2 -> 18.0 | Helped |
| add superior grounding-token selection (GTI) (ablation) | LLaVA-Next-Video · HC-STVGv1 | m_vIoU: 15.2 -> 16.3 | Helped |
| add spatial prompt learning (ablation) | LLaVA-Next-Video with GTI · HC-STVGv1 | m_vIoU: 16.3 -> 18.0 | Helped |
| add temporal prompt learning (ablation) | LLaVA-Next-Video with GTI · HC-STVGv1 | m_vIoU: 16.3 -> 18.4 | Helped |
| add spatial and temporal prompt learning (ablation) | LLaVA-Next-Video with GTI · HC-STVGv1 | m_vIoU: 16.3 -> 19.9 | Helped |
| add temporal-augmented assembling (TAS) (ablation) | LLaVA-Next-Video with GTI and both prompts · HC-STVGv1 | m_vIoU: 19.9 -> 20.4 | Helped |
| add superior grounding-token selection (GTI) (ablation) | LLaVA-OneVision · HC-STVGv1 | m_vIoU: 21.3 -> 23.3 | Helped |
| add spatial prompt learning (ablation) | LLaVA-OneVision with GTI · HC-STVGv1 | m_vIoU: 23.3 -> 24.1 | Helped |
| add temporal prompt learning (ablation) | LLaVA-OneVision with GTI · HC-STVGv1 | m_vIoU: 23.3 -> 23.8 | Helped |
| add spatial and temporal prompt learning (ablation) | LLaVA-OneVision with GTI · HC-STVGv1 | m_vIoU: 23.3 -> 24.3 | Helped |
Surprising findings
- The fixed last special token commonly used in prior work does not consistently give the best grounding; the best token varies across samples and MLLMs.
- Temporal inconsistency under frame reversal correlates strongly with worse spatial grounding accuracy.
Limitations
- Struggles with long videos due to high computational cost of MLLMs.
- Relies on external proposal generators (Grounding DINO, SAM2) and GPT-4o for query decomposition.
- Gap to fully-supervised methods remains on some metrics.
Experimental setup
Datasets: HC-STVGv1 HC-STVGv2 VidSTG RefCOCOg
Benchmarks: HCSTVG-v1 HCSTVG-v2 VidSTG
Metrics: m_vIoU vIoU@0.3 vIoU@0.5 Acc@0.5
Baselines: E3M ReCLIP RedCircle CoSPaL STPro VEM WINNER TubeDETR STCAT CSDVL CG-STVG
Builds on: E3M ControlMLLM TPT SAM2 Grounding DINO
Closest papers by meaning
- STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding (CVPR 2025 · similarity 0.92)
- VideoGEM: Training-free Action Grounding in Videos (CVPR 2025 · similarity 0.91)
- VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.90)
- Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Grounding (ICLR 2025 · similarity 0.90)
- Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding (ICLR 2025 · similarity 0.90)
- Context-Guided Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.