Research map / Video / Action recognition
What When and Where? Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a self-supervised framework that learns spatio-temporal action grounding from narrated HowTo100M videos using Sinkhorn-based frame selection plus joint global and local contrastive losses, and introduces the GroundingYoutube benchmark for untrimmed multi-action videos.
Problem
Localizing actions in space and time in long, untrimmed multi-action instructional videos from text queries, without human bounding-box or sentence annotation, using only noisy ASR narrations.
Approach
Combines a global representation (for temporal extent) and a local token-level cross-attention representation (for spatial localization), trained with NCE losses on frames selected by Sinkhorn optimal transport matching words to frames, so that noisy ASR-paired clips yield better supervision.
Key results
- On GroundingYouTube spatio-temporal grounding, the proposed CLIP-based model reaches 28.26 mAP (0.1:0.5) versus 22.33 for the MIL-NCE+RegionCLIP combination.
- On YouCook-Interactions spatial grounding, the CLIP model achieves 58.35 pointing-game accuracy, outperforming weakly supervised GLIP (52.84) and RegionCLIP (51.56).
- On MiningYoutube temporal grounding, the CLIP model reaches 20.33 IoU versus 18.69 for MIL-NCE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed global-local method instead of MIL-NCE+RegionCLIP | Spatio-temporal grounding · GroundingYouTube full videos | mAP: 22.33 -> 28.26 | Helped |
| Use the proposed method instead of the strongest listed baseline | Spatial grounding · YouCook-Interactions | accuracy: 48.63 -> 58.35 | Helped |
| Use the proposed method instead of STCAT | Spatial grounding · GroundingYoutube | accuracy: 55.90 -> 56.98 | Helped |
| Use the proposed method instead of GLIP | Spatial grounding · V-HICO | accuracy: 66.05 -> 62.34 | Hurt |
| Use the proposed method instead of STCAT | Spatial grounding · Daly | accuracy: 85.42 -> 71.35 | Hurt |
| Use the proposed method instead of MIL-NCE | Temporal grounding · MiningYoutube | IoU: 18.69 -> 20.33 | Helped |
| Add the global loss to the local loss (ablation) | Ours · GroundingYouTube | mAP IoU@0.4: 7.29 -> 19.45 | Helped |
| Add the local loss to the global loss (ablation) | Ours · GroundingYouTube | mAP IoU@0.4: 9.28 -> 19.45 | Helped |
| Add the global loss to the local loss (ablation) | Ours · MiningYoutube | IoU: 5.23 -> 20.33 | Helped |
| Add the local loss to the global loss (ablation) | Ours · MiningYoutube | IoU: 19.12 -> 20.33 | Helped |
| Add the global loss to the local loss (ablation) | Ours · YouCook-Interactions | accuracy: 55.29 -> 58.35 | Helped |
| Add the local loss to the global loss (ablation) | Ours · YouCook-Interactions | accuracy: 36.23 -> 58.35 | Helped |
Surprising findings
- Models trained with only a global loss perform well temporally but poorly spatially, while only-local loss fails temporally, indicating both losses are needed.
- Fully supervised TubeDETR and STCAT are outperformed by the self-supervised model on the instructional-domain spatial benchmarks.
Limitations
- ASR narrations are noisy and loosely aligned, so frame selection is needed and not all sentences refer to visible events.
- Evaluation focuses on cooking and instructional domains; generalization is shown only on limited general-domain datasets.
Experimental setup
Datasets: HowTo100M MiningYoutube YouCook-Interactions V-HICO Daly Vid-STG
Benchmarks: GroundingYoutube MiningYoutube YouCook-Interactions V-HICO Daly
Metrics: IoU+pointing game video mAP pointing game accuracy mAP IoU IoD
Baselines: MIL-NCE CoMMA GLIP RegionCLIP CLIP MIL-NCE+RegionCLIP TubeDETR STCAT MLP (MiningYouTube)
Builds on: MIL-NCE CLIP Sinkhorn-Knopp optimal transport RegionCLIP MMAction-style cross-attention grounding (Attention Rollout)
Closest papers by meaning
- Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding (ICLR 2025 · similarity 0.90)
- STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding (CVPR 2025 · similarity 0.90)
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.89)
- VideoGEM: Training-free Action Grounding in Videos (CVPR 2025 · similarity 0.89)
- VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.88)
- CoSTA: End-to-End Comprehensive Space-Time Entanglement for Spatio-Temporal Video Grounding (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.