Research map / Video / Video-language understanding
Kronecker Mask and Interpretive Prompts are Language-Action Video Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts CLIP to video action recognition by combining a Kronecker-mask temporal transformer (KMTA/KMCTA) on frame features with LLM-generated interpretive action prompts that shift alignment from nouns to verbs.
Problem
Adapting CLIP to video action recognition, where existing methods adjust only the visual or textual branch, temporal modeling is limited in receptive field and suffers spatiotemporal homogenization, and text descriptions emphasize nouns over verbs.
Approach
Introduces Kronecker mask attention, a temporal attention where each token attends to all tokens from other frames (plus a causal variant), implemented by masking joint attention, and augments text with LLM-generated sentence-level interpretive prompts (action decomposition, synonyms, body parts) so the model focuses on verbs.
Key results
- CLAVER-B/16 (KMT, 16 frames) reaches 85.9% top-1 on Kinetics-400 and CLAVER-L/14 (336, KMCT, 16 frames) reaches 88.9%.
- Zero-shot CLAVER (KMT) achieves 54.1% on HMDB-51 and 78.6% on UCF-101, and 73.8% top-1 on Kinetics-600 zero-shot.
- On the CLIP-B/32 ablation, KMTA at 1/3 scale plus interpretive prompts improves top-1 from 77.4% (CLIP-Mean) to 81.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add KMTA at 1/3 scale (ablation) | CLIP-Mean on Kinetics-400 · Kinetics-400 | Top-1 accuracy: 77.4 -> 80.9 | Helped |
| add interpretive prompts (ablation) | CLIP-Mean on Kinetics-400 · Kinetics-400 | Top-1 accuracy: 77.4 -> 78.4 | Helped |
| combine KMTA at 1/3 scale with interpretive prompts (ablation) | CLIP-Mean on Kinetics-400 · Kinetics-400 | Top-1 accuracy: 77.4 -> 81.5 | Helped |
| increase KMTA depth from 1 layer to 1/6 and 1/3 scale (ablation) | CLIP-Mean on Kinetics-400 · Kinetics-400 | Top-1 accuracy: 78.3 -> 79.6 -> 80.9 | Helped |
| replace mean pooling with class-token-only temporal attention (ablation) | CLAVER-B/32 · Kinetics-400 | Top-1 accuracy: 78.4 -> 78.9 | Helped |
| replace mean pooling with pipeline temporal attention (ablation) | CLAVER-B/32 · Kinetics-400 | Top-1 accuracy: 78.4 -> 79.4 | Helped |
| replace joint attention with KMTA (ablation) | CLAVER-B/32 · Kinetics-400 | Top-1 accuracy: 80.1 -> 81.5 | Helped |
| replace KMTA with KMCTA (ablation) | CLAVER-B/32 · Kinetics-400 | Top-1 accuracy: 81.5 -> 81.4 | Hurt |
| add prefix-suffix templates (ablation) | action noun and phrase prompts · Kinetics-400 | Top-1 accuracy: 77.4 -> 77.4 | No clear effect |
| add action-decomposition interpretive prompts (ablation) | prefix-suffix action prompts · Kinetics-400 | Top-1 accuracy: 77.4 -> 78.0 | Helped |
| add synonym-conversion interpretive prompts (ablation) | action-decomposition prompts · Kinetics-400 | Top-1 accuracy: 78.0 -> 78.2 | Helped |
| add prompts involving body parts (ablation) | synonym-conversion prompts · Kinetics-400 | Top-1 accuracy: 78.2 -> 78.4 | Helped |
Surprising findings
- Token shuffling after adding time embedding barely changes joint attention predictions, which the authors interpret as spatiotemporal homogenization; KMTA/KMCTA are much more sensitive to shuffling (e.g., 81.5% to 43.5%).
Limitations
- Image encoder frozen in main experiments; gains from fine-tuning both branches are shown only in ablation.
- Lower accuracy than methods using longer frames, larger pretraining data, or higher resolution (e.g., MTV-H, ViViT-H).
Experimental setup
Datasets: Kinetics-400 Kinetics-600 UCF-101 HMDB-51
Benchmarks: Kinetics-400 Kinetics-600 UCF-101 HMDB-51
Metrics: Top-1 accuracy Top-5 accuracy
Baselines: X-CLIP ILA ASU MAXI OST EVL ViFiCLIP ActionCLIP Swin-L MViTv2-L MTV-H CoVeR
Builds on: CLIP X-CLIP ActionCLIP ViT Bhojanapalli low-rank softmax theorem
Closest papers by meaning
- BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition (ICML 2025 · similarity 0.89)
- Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP (AAAI 2025 · similarity 0.89)
- STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding (CVPR 2025 · similarity 0.88)
- Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Efficient Transfer Learning for Video-language Foundation Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.