Research map / Video / Video-language understanding
VicTR: Video-conditioned Text Representations for Activity Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts pretrained CLIP to video by making text embeddings video-conditioned (token-boosting, cross-modal and temporal attention, affinity re-weighting) and optionally using visually-grounded auxiliary text, achieving strong few-shot, zero-shot, short-form and long-form activity recognition.
Problem
Adapting image-text VLMs to video recognition has focused on temporally augmenting visual embeddings while keeping text embeddings frozen, discarded, or static across videos, limiting alignment flexibility in the contrastive latent space.
Approach
Generate per-video, per-frame text tokens by weighting class-text embeddings with frame-text affinities, jointly process them with visual tokens via divided cross-modal and temporal attention, and re-weight text with affinities in each layer, optionally adding auxiliary semantic text (objects, scenes, human subjects) as visually-grounded supervision.
Key results
- VicTR (B/16) reaches 51.0% zero-shot on HMDB-51 and 72.4% on UCF-101, outperforming X-CLIP by 6.4 and 0.4 points.
- VicTR (L/14) achieves 87.0% top-1 on Kinetics-400 with 8 frames and 4x3 views at 656 GFLOPs per view, competitive with video-VLMs.
- On Charades, VicTR (B/16) reaches 50.1 mAP, +5.2 over CLIP Hitchhiker's (44.9) and +5.5 over ActionCLIP (44.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add auxiliary text (ablation) | VicTR · Kinetics-400 | top-1 accuracy (%): 84.2 -> 84.4 | Helped |
| add auxiliary text (ablation) | VicTR · Charades | mAP (%): 49.8 -> 50.1 | Helped |
| replace CLIP visual embeddings with VicTR video embeddings (ablation) | VicTR · Kinetics-400 | top-1 accuracy (%): 84.0 -> 84.4 | Helped |
| replace CLIP visual embeddings with VicTR video embeddings (ablation) | VicTR · Charades | mAP (%): 49.7 -> 50.1 | Helped |
| use video-conditioned text embeddings instead of CLIP text embeddings (ablation) | VicTR · Kinetics-400 | top-1 accuracy (%): 83.3 -> 84.4 | Helped |
| use video-conditioned text embeddings instead of CLIP text embeddings (ablation) | VicTR · Charades | mAP (%): 41.7 -> 50.1 | Helped |
| add affinity weighting (ablation) | VicTR · Charades | mAP (%): 48.8 -> 50.1 | Helped |
| use divided attention instead of joint attention (ablation) | VicTR · Charades | mAP (%): 44.8 -> 50.1 | Helped |
| use affinity-based logits instead of text-only logits (ablation) | VicTR · Charades | mAP (%): 41.2 -> 50.1 | Helped |
| use affinity-based logits instead of visual-only logits (ablation) | VicTR · Charades | mAP (%): 43.1 -> 50.1 | Helped |
| use VicTR instead of X-CLIP | video-VLMs · HMDB-51 zero-shot | top-1 accuracy (%): 44.6 ± 5.2 -> 51.0 ± 1.3 | Helped |
| use VicTR instead of X-CLIP | video-VLMs · UCF-101 zero-shot | top-1 accuracy (%): 72.0 ± 2.3 -> 72.4 ± 0.3 | Helped |
Surprising findings
- Replacing video embeddings with temporally pooled CLIP frame embeddings drops performance only slightly (-0.4 on Kinetics-400 and -0.4 mAP on Charades), while replacing video-conditioned text with original CLIP text drops it substantially (-8.4 mAP on Charades).
- Divided attention outperforms joint attention by 5.3 mAP on Charades, suggesting joint attention is harder to train.
Limitations
- Auxiliary text gains are small (+0.2 on Kinetics-400, +0.3 mAP on Charades), and the auxiliary semantics lack ground-truth annotations.
- Absolute performance may not be state-of-the-art on some settings such as long-form recognition, as the authors note.
Experimental setup
Datasets: Kinetics-400 HMDB-51 UCF-101 Charades
Benchmarks: HMDB-51 UCF-101 Kinetics-400 Charades
Metrics: top-1 accuracy mean average precision
Baselines: X-CLIP X-Florence ActionCLIP CLIP Hitchhiker's EVL CLIP4clip Video-Swin TimeSformer TSM
Builds on: CLIP X-CLIP ActionCLIP EVL TimeSformer
Closest papers by meaning
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.88)
- OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition (CVPR 2024 · similarity 0.86)
- ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models (ICLR 2024 · similarity 0.86)
- Align Before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition (CVPR 2024 · similarity 0.86)
- Bridging Vision and Language Spaces with Assignment Prediction (ICLR 2024 · similarity 0.86)
- InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.