Research map / Video / Video-language understanding
STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STOP, a video prompting method for frozen CLIP that adds intra-frame spatial prompts at discriminative regions and inter-frame temporal prompts inserted between frames with high temporal variation, evaluated on action recognition and video-text retrieval.
Problem
Adapting frozen vision-language models like CLIP to video tasks with static video prompts that ignore per-video temporal dynamics and spatial variation across frames.
Approach
Dynamically generated prompts: intra-frame prompts placed at regions identified by combining CLIP attention with 3D-conv temporal dynamics, and inter-frame prompts whose count scales with measured frame-to-frame variation, so prompts adapt to each video rather than being static.
Key results
- On HMDB51, UCF101 and SSV2 STOP reaches 72.0%, 95.3% and 21.4% top-1, improving over the best prior prompt method by 2.2, 1.7 and 3.1 points.
- On MSR-VTT text-to-video, STOP reaches R@1 47.7 vs 46.3 for MPT-Transformer while using 7.53MB trainable parameters versus 9.61MB.
- Ablation on HMDB51 shows intra-frame prompts alone add 16.9 points and inter-frame prompts alone add 19.2 points over the frozen CLIP baseline, with both giving 72.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use STOP instead of CLIP4Clip | CLIP-ViT-B/32 video action recognition · HMDB51 | ACC@1: 75.2 -> 72.0 | Hurt |
| use STOP instead of DGL-Transformer | CLIP-ViT-B/32 video action recognition · UCF101 | ACC@1: 93.6 -> 95.3 | Helped |
| use STOP instead of DGL-Linear | CLIP-ViT-B/32 video action recognition · SS-V2 | ACC@1: 18.3 -> 21.4 | Helped |
| use STOP instead of MPT-Transformer | CLIP-ViT-B/32 video-text retrieval · MSR-VTT text-to-video | R@1: 46.3 -> 47.7 | Helped |
| use STOP instead of MPT-Transformer | CLIP-ViT-B/32 video-text retrieval · MSR-VTT video-to-text | MnR: 12.7 -> 12.2 | Helped |
| add intra-frame spatial prompting (ablation) | STOP · HMDB51 | ACC@1: 50.2 -> 67.1 | Helped |
| add intra-frame spatial prompting (ablation) | STOP · UCF101 | ACC@1: 70.2 -> 83.1 | Helped |
| add intra-frame spatial prompting (ablation) | STOP · SS-V2 | ACC@1: 10.2 -> 16.5 | Helped |
| add inter-frame temporal prompting (ablation) | STOP · HMDB51 | ACC@1: 50.2 -> 69.4 | Helped |
| add inter-frame temporal prompting (ablation) | STOP · UCF101 | ACC@1: 70.2 -> 87.4 | Helped |
| add inter-frame temporal prompting (ablation) | STOP · SS-V2 | ACC@1: 10.2 -> 15.9 | Helped |
| add inter-frame temporal prompting to intra-frame prompting (ablation) | STOP · HMDB51 | ACC@1: 67.1 -> 72.0 | Helped |
Limitations
- Relies on a fixed hyperparameter set (number of prompts, scaling factors alpha/beta/eta) tuned via sensitivity analysis on HMDB51.
- Evaluated only with CLIP ViT-B/32 backbone and 12 or 64 sampled frames.
Experimental setup
Datasets: HMDB51 UCF101 SSV2 MSR-VTT ActivityNet DiDeMo VATEX
Benchmarks: HMDB51 UCF101 SSV2 MSR-VTT ActivityNet DiDeMo VATEX
Metrics: ACC@1 R@1 R@5 R@10 MnR
Baselines: CLIP4Clip AdapterBias AdapterATTN AdapterFFN Visual-Text Adapter Video-Text Adapter Efficient Prompt VPT UPT VoPF+C DGL-Linear DGL-Transformer
Builds on: CLIP4Clip VoP DGL MPT VPT
Closest papers by meaning
- State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understanding (NeurIPS 2025 · similarity 0.91)
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.89)
- Kronecker Mask and Interpretive Prompts are Language-Action Video Learners (ICLR 2025 · similarity 0.88)
- Visual In-Context Prompting (CVPR 2024 · similarity 0.88)
- VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM (CVPR 2025 · similarity 0.88)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.