Research map / Video / Video-language understanding
VideoGEM: Training-free Action Grounding in Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoGEM, a training-free method that adapts GEM's self-self attention to video-language models and adds static/dynamic layer weighting and verb/object/action prompt decomposition for spatial action grounding in videos.
Problem
Spatially localizing actions in videos with vision-language models without bounding-box training; actions lack clear physical outlines and are described by high-level concepts, and existing video grounding methods require training.
Approach
Extends the training-free GEM self-self attention to video backbones, weights higher transformer layers via a mix of static and prompt-dependent dynamic weights, and decomposes action prompts into verb, object, and action prompts whose predicted centers are averaged.
Key results
- VideoGEM with ViCLIP achieves 66.58% average accuracy across V-HICO, DALY, YouCook-Interactions and GroundingYouTube, outperforming the best trained baseline WWW-CLIP (62.26% average) by more than 3 points.
- Prompt decomposition improves over the best single-prompt mode by over 5% average accuracy across backbones (e.g., ViCLIP 60.95 action-only vs 66.58 combined).
- Static plus dynamic layer weighting improves over no weighting by about 1% average accuracy for both ViCLIP and OpenCLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VideoGEM's layer weighting and prompt decomposition to GEM | CLIP · V-HICO | accuracy: 67.79 -> 76.90 | Helped |
| Add VideoGEM's layer weighting and prompt decomposition to GEM | OpenCLIP · YouCook-Interactions | accuracy: 56.87 -> 60.05 | Helped |
| Add VideoGEM's layer weighting and prompt decomposition to GEM | ViCLIP · GroundingYouTube | accuracy: 51.28 -> 57.21 | Helped |
| Add static layer weights (ablation) | VideoGEM with ViCLIP · V-HICO | accuracy: 74.79 -> 76.18 | Helped |
| Add dynamic layer weights (ablation) | VideoGEM with OpenCLIP · GroundingYouTube | accuracy: 40.17 -> 43.29 | Helped |
| Use combined static and dynamic layer weights (ablation) | VideoGEM with OpenCLIP · GroundingYouTube | accuracy: 40.17 -> 45.33 | Helped |
| Use prompt decomposition instead of only an action prompt (ablation) | VideoGEM with ViCLIP · GroundingYouTube | accuracy: 51.99 -> 57.21 | Helped |
| Use prompt decomposition instead of only an action prompt (ablation) | VideoGEM with OpenCLIP · YouCook-Interactions | accuracy: 57.36 -> 60.05 | Helped |
| Combine prompt predictions by weighted averaging predicted positions rather than averaging heatmaps (ablation) | VideoGEM with ViCLIP · GroundingYouTube | accuracy: 49.96 -> 57.21 | Helped |
| Use video frames rather than a repeated image (ablation) | VideoGEM with ViCLIP · GroundingYouTube | accuracy: 55.08 -> 57.21 | Helped |
| Use video frames rather than a repeated image (ablation) | VideoGEM with ViCLIP · DALY | accuracy: 78.47 -> 78.25 | No clear effect |
Surprising findings
- Using only the first few GEM layers rather than more layers does not help; accuracy increases with more GEM layers, saturating at seven, while removing final layers hurts most.
- Video input gives only about 1% average gain over repeating a single image eight times, with little benefit on V-HICO and DALY where actions are more static.
Limitations
- Relies on pretrained vision-language backbones whose object bias affects verb-object prompts.
- Evaluated only on four datasets; GroundingYouTube and YouCook-Interactions are restricted to cooking domain.
- Verb and object extraction for natural-language annotations depends on an NLP tool.
Experimental setup
Datasets: V-HICO DALY YouCook-Interactions GroundingYouTube
Benchmarks: V-HICO DALY YouCook-Interactions GroundingYouTube
Metrics: accuracy
Baselines: TubeDETR STCAT VideoGrounding-DINO GLIP CoMMA RegionCLIP WWW-CLIP GEM
Builds on: GEM CLIPSurgery MaskCLIP ViCLIP
Closest papers by meaning
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.91)
- VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.90)
- Comprehensive Visual Grounding for Video Description (AAAI 2024 · similarity 0.90)
- What When and Where? Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions (CVPR 2024 · similarity 0.89)
- STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding (CVPR 2025 · similarity 0.89)
- Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.