Research map / Video / Action recognition
Does Video-Text Pretraining Help Open-Vocabulary Online Action Detection?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OV-OAD, a zero-shot online action detector trained only on video-text pairs (InternVid-5K and ActivityNet) with an object-centric decoder that clusters frames, evaluated zero-shot on THUMOS'14, TVSeries, FineAction and EK100.
Problem
Online action detection is closed-set and needs frame-level annotations, limiting open-world real-time use; the paper asks whether video-text pretraining can enable zero-shot open-vocabulary online action detection.
Approach
A visual-text dual-encoder with a distant neighboring-frame transformer and an object-centric slot-attention clustering block, trained with current frame-text contrast, multi-label video-text contrast and background mask prediction, using only text supervision and no frame-level labels.
Key results
- OV-OAD achieves 37.5% mAP on THUMOS'14 zero-shot (ANet pretraining), vs 28.0-30.9% for CLIP-based baselines.
- OV-OAD achieves 73.8% cAP on TVSeries with InternVid-5K pretraining, vs 71.6% for CLIP-III.
- OV-OAD reaches 292.3 FPS inference (feature extraction bottleneck) versus LSTR's optical flow bottleneck at 8.1 FPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain OV-OAD on ANet rather than use CLIP-III frame clustering | Zero-shot online action detection · THUMOS’14 | mAP: 29.7 -> 37.5 | Helped |
| Pretrain OV-OAD on InternVid-5K rather than use CLIP-III frame clustering | Zero-shot online action detection · TVSeries | cAP: 71.6 -> 73.8 | Helped |
| Add current frame-text contrastive loss (ablation) | OV-OAD with multi-label video-text contrastive loss · THUMOS’14 | mAP: 32.9 -> 36.3 | Helped |
| Add background mask loss (ablation) | OV-OAD with multi-label video-text contrastive loss · THUMOS’14 | mAP: 32.9 -> 33.6 | Helped |
| Add both current frame-text contrastive loss and background mask loss (ablation) | OV-OAD with multi-label video-text contrastive loss · THUMOS’14 | mAP: 32.9 -> 37.5 | Helped |
| Remove the distant neighboring-frame Transformer block (ablation) | OV-OAD · THUMOS’14 | mAP: 37.5 -> 33.3 | Hurt |
| Replace Transformer Decoder ΨDNTR with a 4-layer Transformer Encoder (ablation) | OV-OAD · THUMOS’14 | mAP: 37.5 -> 35.6 | Hurt |
| Add cross-attention after the 4-layer Transformer Encoder ΨDNTR (ablation) | Encoder-based ΨDNTR · THUMOS’14 | mAP: 35.6 -> 36.5 | Helped |
| Replace the object-centric decoder with a Transformer Decoder in ΨAC (ablation) | OV-OAD · THUMOS’14 | mAP: 37.5 -> 37.0 | Hurt |
| Remove the final Transformer encoder in ΨAC (ablation) | OV-OAD · THUMOS’14 | mAP: 37.5 -> 36.9 | Hurt |
| Freeze the full pretrained text encoder (ablation) | OV-OAD text encoder · THUMOS’14 | mAP: 37.5 -> 35.8 | Hurt |
| Add Adapter while continuing to train the text encoder backbone (ablation) | OV-OAD text encoder · THUMOS’14 | mAP: 36.7 -> 37.5 | Helped |
Surprising findings
- Base-to-novel fine-tuning of OadTR, LSTR and MAT with text encoders performs poorly on unseen categories (e.g., 9.6 mAP for OadTR at 50/50 split).
- Directly using ViCLIP, a video-text model, as a visual extractor yields unsatisfactory zero-shot online detection performance.
Limitations
- Visual encoder uses image-only CLIP features, so it underweights spatio-temporal motion structure.
- Poor performance on fine-grained actions with low foreground and similar backgrounds (e.g., CliffDiving vs CliffShot, BaseballPitch 13.17 AP).
- Pretraining data noise from auto-generated captions; InternVid subset is only 5K videos.
Experimental setup
Datasets: InternVid-5K ActivityNet v1.3 THUMOS'14 TVSeries FineAction EPIC-KITCHENS-100
Benchmarks: THUMOS'14 TVSeries FineAction EPIC-KITCHENS-100
Metrics: mAP cAP
Baselines: CLIP-I CLIP-II CLIP-III OadTR LSTR MAT ViCLIP
Builds on: CLIP AdaptFormer GroupViT slot attention OadTR LSTR MAT
Closest papers by meaning
- Open-Vocabulary Video Relation Extraction (AAAI 2024 · similarity 0.88)
- Text-Infused Attention and Foreground-Aware Modeling for Zero-Shot Temporal Action Detection (NeurIPS 2024 · similarity 0.88)
- Context-Enhanced Memory-Refined Transformer for Online Action Detection (CVPR 2025 · similarity 0.87)
- OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition (CVPR 2024 · similarity 0.87)
- Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP (AAAI 2025 · similarity 0.87)
- Kronecker Mask and Interpretive Prompts are Language-Action Video Learners (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.