Research map / Video / Action recognition
OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Augments action class names with GPT-3.5-generated spatial and temporal descriptors and matches them to video frames via an entropy-regularized optimal transport solver, adapting CLIP for zero-shot, few-shot, and fully supervised video recognition.
Problem
Adapting image-language models like CLIP to video recognition is limited because concise action category names have low semantic distinctiveness compared with web-scale descriptive narratives, and pooled descriptor matching is suboptimal and sensitive to LLM hallucination.
Approach
Disentangle category names into Spatio Descriptors (static scene/objects) and Temporal Descriptors (step decomposition) generated by an LLM, then match frame-level features to these descriptors via a Sinkhorn optimal transport plan (Optimal Descriptor Solver) instead of simple pooling.
Key results
- Zero-shot Kinetics-600 Top-1 accuracy of 75.1% with 32 frames, versus 71.5% for MAXI and 71.2% for ViFi-CLIP.
- Few-shot HMDB-51 with K=2 under fine-tuning on K400 reaches 64.8% vs 58.0% for MAXI at K=2.
- Fully supervised gains of 0.2 to 0.8 points over Text4Vis across Kinetics-400, SSv2, and ActivityNet for ViT-B/32 and B/16.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace category-name matching with pooled descriptors (ablation) | OST zero-shot video recognition · HMDB | Top-1 accuracy: 50.9 -> 53.3 (+2.4) | Helped |
| Replace category-name matching with pooled descriptors (ablation) | OST zero-shot video recognition · UCF | Top-1 accuracy: 75.5 -> 76.6 (+1.1) | Helped |
| Replace category-name matching with pooled descriptors (ablation) | OST zero-shot video recognition · Kinetics-600 | Top-1 accuracy: 70.8 -> 69.3 | Hurt |
| Use Optimal Descriptor Solver instead of pooled descriptor matching (ablation) | OST zero-shot video recognition · HMDB | Top-1 accuracy: 50.9 -> 54.5 (+3.6) | Helped |
| Use Optimal Descriptor Solver instead of pooled descriptor matching (ablation) | OST zero-shot video recognition · UCF | Top-1 accuracy: 75.5 -> 77.9 (+2.4) | Helped |
| Use Optimal Descriptor Solver instead of pooled descriptor matching (ablation) | OST zero-shot video recognition · Kinetics-600 | Top-1 accuracy: 70.8 -> 72.3 (+1.5) | Helped |
| Use Spatio descriptors only (ablation) | OST zero-shot video recognition · HMDB | Top-1 accuracy: 54.5 -> 46.7 | Hurt |
| Use Temporal descriptors only (ablation) | OST zero-shot video recognition · HMDB | Top-1 accuracy: 54.5 -> 53.1 | Hurt |
| Use 2 instead of 4 Spatio-Temporal descriptors (ablation) | OST zero-shot video recognition · HMDB | Top-1 accuracy: 54.5 -> 53.8 | Hurt |
| Use 8 instead of 4 Spatio-Temporal descriptors (ablation) | OST zero-shot video recognition · HMDB | Top-1 accuracy: 54.5 -> 53.0 | Hurt |
| Condition Temporal descriptors on category names (ablation) | OST zero-shot video recognition · HMDB | Top-1 accuracy: 49.8 -> 54.5 | Helped |
| Condition both Spatio and Temporal descriptors on category names (ablation) | OST zero-shot video recognition · UCF | Top-1 accuracy: 79.0 -> 78.9 | Hurt |
Surprising findings
- Simply mean-pooling descriptors with category names lowers K600 zero-shot accuracy by 1.5 points, while the OD Solver recovers and improves it.
- Conditioning both Spatio and Temporal descriptors on category names degrades performance; conditioning only temporal descriptors works best.
Limitations
- LLM hallucination can introduce noisy descriptors, especially with more descriptors (8).
- Descriptors are generated by a single GPT-3.5 query with a fixed prompt and temperature, with no evaluation of alternative LLMs stated in the main text.
Experimental setup
Datasets: Kinetics-400 Kinetics-600 UCF-101 HMDB-51 Something-Something V2 ActivityNet ImageNet
Benchmarks: HMDB-51 UCF-101 Kinetics-600 Kinetics-400 Something-Something V2 ActivityNet
Metrics: Top-1 accuracy Top-5 accuracy throughput training latency
Baselines: ViFi-CLIP MAXI XCLIP ActionCLIP A5 DiST Vita-CLIP Text4Vis ER-ZSAR JigsawNet
Builds on: CLIP ViFi-CLIP Text4Vis Sinkhorn-Knopp algorithm CuPL
Closest papers by meaning
- Efficient Transfer Learning for Video-language Foundation Models (CVPR 2025 · similarity 0.89)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.88)
- MoTE: Reconciling Generalization with Specialization for Visual-Language to Video Knowledge Transfer (NeurIPS 2024 · similarity 0.87)
- Storyboard-guided Alignment for Fine-grained Video Action Recognition (NeurIPS 2025 · similarity 0.87)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.87)
- Enhanced Motion-Text Alignment for Image-to-Video Transfer Learning (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.