Research map / Video / Action recognition
Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STDD, a CLIP-based zero-shot action recognition framework with a parameter-free Space-time Cross Attention (window shift masking + multi-scale channel mixing) and LLM-built action semantic knowledge graph for spatiotemporal text prompts.
Problem
Zero-shot action recognition with CLIP struggles to capture temporal dynamics in vision and fine-grained spatiotemporal semantics in text, especially for novel actions with subtle differences.
Approach
Transforms spatial self-attention into Space-time Cross Attention via masking, channel mixing across interleaved tokens at multiple time scales, padding, and short-cutting, adding no parameters; and augments text with an LLM-built Action Semantic Knowledge Graph split into static (spatial) and dynamic (temporal) prompts, aligned frame-to-prompt with distillation from frozen CLIP.
Key results
- On UCF101 EP 2 with ViT-L/14, the method reaches 88.6% vs. 87.9% for Open-VCLIP; on EP 3 with ViT-B/16 it reaches 85.2% UCF, 55.9% HMDB and 75.1% K600, surpassing the prior best per dataset.
- Space-time Cross Attention with fully-tuned adaptation improves UCF accuracy from 74.2% (frozen CLIP) to 85.2% (+11.0%) without extra parameters.
- Spatiotemporal text prompts improve over spatial-only prompts by 1.08% for the proposed model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STDD instead of Open-VCLIP | CLIP ViT-L/14 · UCF101 EP 1 and EP 2 | accuracy: 93.1 ± 1.9 -> 93.4 ± 2.2; 87.9 -> 88.6 | Helped |
| Use STDD instead of Open-VCLIP | CLIP ViT-L/14 · HMDB51 EP 1 and EP 2 | accuracy: 68.5 ± 4.0 -> 68.7 ± 4.5; 58.3 -> 58.7 | Helped |
| Use STDD instead of the strongest listed baseline | CLIP ViT-B/16 · UCF101 EP 3 | accuracy: 84.8 ± 1.1 -> 85.2 ± 1.2 | Helped |
| Use STDD instead of the strongest listed baseline | CLIP ViT-B/16 · HMDB51 EP 3 | accuracy: 55.6 ± 1.4 -> 55.9 ± 0.2 | Helped |
| Use STDD instead of the strongest listed baseline | CLIP ViT-B/16 · Kinetics-600 | accuracy: 74.8 ± 0.9 -> 75.1 ± 0.7 | Helped |
| Add MCM to fully tuned adaptation (ablation) | STDD · UCF101 EP 3 | accuracy: 74.2 -> 84.5 | Helped |
| Add WSM alongside MCM (ablation) | Fully tuned STDD with separate MCM · UCF101 EP 3 | accuracy: 84.5 -> 85.0 | Helped |
| Use continual rather than separate MCM (ablation) | Fully tuned STDD with WSM · UCF101 EP 3 | accuracy: 85.0 -> 85.2 | Helped |
| Use fully tuned rather than prompt-based adaptation (ablation) | STDD with WSM and continual MCM · UCF101 EP 3 | accuracy: 82.7 -> 85.2 | Helped |
| Add temporal shift to random spatial masking (ablation) | STDD · UCF101 EP 3 | accuracy: 79.6% vs. 83.7% | Helped |
| Use repeat-window rather than random-window masking (ablation) | STDD with temporal shift · UCF101 EP 3 and HMDB51 EP 3 | accuracy: -0.1% and -0.3% | Helped |
| Set WSM mask ratio to 50% rather than 25% or 75% (ablation) | STDD · UCF101 EP 3 and HMDB51 EP 3 | accuracy: 85.2 vs. 85.0; 55.9 vs. 55.5; 55.9 vs. 55.7 | Helped |
Surprising findings
- Spatiotemporal text prompts give negligible gain over spatial prompts for vanilla CLIP, suggesting object bias in CLIP.
- Average pooling of frame-level features reduces accuracy for CLIP with category prompts (-0.6% / -0.4%), suggesting frame-level alignment matters.
Limitations
- LLM hallucination makes quality of generated spatiotemporal prompts vary across categories.
- Purely text-based augmentation lacks visual cues, which may limit gains.
- Evaluated only on cross-dataset zero-shot protocols with K400 training.
Experimental setup
Datasets: Kinetics-400 GPT-3.5-generated action semantic knowledge graph
Benchmarks: UCF101 HMDB51 Kinetics-600
Metrics: top-1 accuracy
Baselines: BIKE Text4Vis Open-VCLIP ViLT-CLIP ActionCLIP X-CLIP MAXI VicTR OST AP-CLIP Open-VCLIP++ FROSTER
Builds on: CLIP X-CLIP Open-VCLIP AIM MAR
Closest papers by meaning
- Kronecker Mask and Interpretive Prompts are Language-Action Video Learners (ICLR 2025 · similarity 0.89)
- Text-Infused Attention and Foreground-Aware Modeling for Zero-Shot Temporal Action Detection (NeurIPS 2024 · similarity 0.88)
- Does Video-Text Pretraining Help Open-Vocabulary Online Action Detection? (NeurIPS 2024 · similarity 0.87)
- OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition (CVPR 2024 · similarity 0.87)
- Efficient Transfer Learning for Video-language Foundation Models (CVPR 2025 · similarity 0.87)
- Test-Time Zero-Shot Temporal Action Localization (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.