Research map / Video / Video-language understanding
Holistic Features are almost Sufficient for Text-to-Video Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TeachCLIP, which distills fine-grained frame-text relevance from heavy CLIP-based T2VR teachers (X-CLIP, TS2-Net, X-Pool, DRL) into a CLIP4Clip student with an attentional frame aggregation block, keeping holistic-feature retrieval cost.
Problem
Fine-grained CLIP-based text-to-video retrieval models are accurate but have high storage and computation overhead, hurting scalability for large-scale retrieval.
Approach
Add an Attentional frame-Feature Aggregation (AFA) block to CLIP4Clip and supervise its frame weights with frame-text relevance scores from a frozen teacher, combined with a Pearson-distance coarse-grained loss, so the student gains fine-grained knowledge while still using single holistic video vectors at retrieval.
Key results
- On MSRVTT-1k, TeachCLIP reaches SumR 203.7 vs 195.5 for CLIP4Clip with the same retrieval cost (ViT-B/32), and the multi-teacher setup reaches 204.6 in ablation.
- Across six datasets TeachCLIP has mean SumR 202.9 versus 197.3 for CLIP4Clip and 203.4 for X-CLIP, and marginally surpasses X-CLIP on MSVD (210.2 vs 210.1) and VATEX (251.6 vs 248.5).
- Using ViT-B/16 as the visual backbone, TeachCLIP lifts SumR on MSRVTT-1k from 201.5 to 207.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| teach CLIP4Clip-AFA with X-CLIP and X-Pool | CLIP4Clip-AFA · MSRVTT-1k | SumR: 196.3 -> 203.7 | Helped |
| teach CLIP4Clip with X-CLIP | CLIP4Clip · MSRVTT-3k | SumR: 150.1 -> 156.0 | Helped |
| teach CLIP4Clip with X-CLIP | CLIP4Clip · MSVD | SumR: 206.6 -> 210.2 | Helped |
| teach CLIP4Clip with X-CLIP | CLIP4Clip · VATEX | SumR: 248.5 -> 251.6 | Helped |
| add AFA to CLIP4Clip (ablation) | CLIP4Clip · MSRVTT-1k | SumR: 195.5 -> 196.3 | Helped |
| add X-CLIP teaching to CLIP4Clip-AFA (ablation) | CLIP4Clip-AFA · MSRVTT-1k | SumR: 196.3 -> 199.8 | Helped |
| add fine-grained teaching with Huber loss (ablation) | CLIP4Clip-AFA with InfoNCE · MSRVTT-1k | SumR: 196.3 -> 196.9 | Helped |
| add fine-grained teaching with Pearson loss (ablation) | CLIP4Clip-AFA with InfoNCE · MSRVTT-1k | SumR: 196.3 -> 197.2 | Helped |
| add fine-grained teaching with cross-entropy loss (ablation) | CLIP4Clip-AFA with InfoNCE · MSRVTT-1k | SumR: 196.3 -> 197.3 | Helped |
| add coarse-grained teaching with Huber loss (ablation) | CLIP4Clip-AFA with InfoNCE · MSRVTT-1k | SumR: 196.3 -> 197.8 | Helped |
| add coarse-grained teaching with Pearson loss (ablation) | CLIP4Clip-AFA with InfoNCE · MSRVTT-1k | SumR: 196.3 -> 198.0 | Helped |
| combine coarse- and fine-grained teaching with Pearson CgT and cross-entropy FgT (ablation) | CLIP4Clip-AFA with InfoNCE · MSRVTT-1k | SumR: 196.3 -> 199.8 | Helped |
Surprising findings
- The student TeachCLIP slightly surpasses its teacher X-CLIP on MSVD and VATEX, which the authors attribute to complementary teacher information acting like ensemble learning.
- Adding an AFA block alone without teaching does not help cross-dataset generalization: CLIP4Clip-AFA transfers worse than CLIP4Clip.
Limitations
- Most experiments use short ~10-second clips and ViT-B/32 backbone, so the claim that holistic features are almost sufficient may not hold for longer videos or stronger backbones.
- Gains from a stronger ViT-B/16 backbone on other datasets are not yet studied.
- Training requires a frozen heavy teacher, adding training-time overhead.
Experimental setup
Datasets: MSRVTT-1k MSRVTT-3k MSVD VATEX ActivityNet Captions DiDeMo
Benchmarks: MSRVTT-1k MSRVTT-3k MSVD VATEX ActivityNet Captions DiDeMo
Metrics: R@1 R@5 R@10 SumR
Baselines: CLIP4Clip X-CLIP TS2-Net X-Pool DRL PromptSwitch CenterCLIP Cap4Video CLIPPING STAN CLIP-ViP UMT
Builds on: CLIP4Clip TeachText X-CLIP CrossKD
Closest papers by meaning
- Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learning (AAAI 2024 · similarity 0.86)
- MUSE: Mamba Is Efficient Multi-scale Learner for Text-video Retrieval (AAAI 2025 · similarity 0.86)
- Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrieval (CVPR 2025 · similarity 0.86)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.85)
- Text Proxy: Decomposing Retrieval from a 1-to-N Relationship into N 1-to-1 Relationships for Text-Video Retrieval (AAAI 2025 · similarity 0.85)
- Learning Dynamic Similarity by Bidirectional Hierarchical Sliding Semantic Probe for Efficient Text Video Retrieval (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.