Research map / Video / Video-language understanding
Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EERCF, a CLIP4Clip-based text-to-video retrieval method with a parameter-free text-gated interaction block for frame- and patch-level features, a Pearson-based intra-feature loss, and a coarse-to-fine recall-then-rerank pipeline.
Problem
Text-to-video retrieval methods with heavy text-video fusion blocks achieve strong accuracy but incur prohibitive computation cost at retrieval time, and text-agnostic global video features over-abstract the content.
Approach
A parameter-free text-gated interaction block (softmax attention with temperature over frame and patch features conditioned on the query sentence) produces multi-granularity video features; coarse text-agnostic features perform top-k recall and fine text-driven features rerank the candidates, with a Pearson constraint added for intra-feature decorrelation.
Key results
- On MSRVTT-1K-Test with ViT-B/32, EERCF reaches 47.8 t2v R@1 and 68.6 t2v mean, matching DRL (68.6) at about 16.0k FLOPs versus DRL's 220.4k.
- With ViT-B/16, EERCF reaches 49.9 t2v R@1 and 70.2 mean at 0.8k FLOPs, comparable to DRL (70.5) at 220.4k FLOPs.
- The paper reports FLOPs reductions of roughly 14x, 39x, 20x and 126x versus the SOTA on MSRVTT-1K, MSRVTT-3K, VATEX and ActivityNet respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the text-patch interaction feature (ablation) | EERCF · MSRVTT-1K-Test t2v | Mean: 67.5 -> 67.8 | Helped |
| add the text-frame interaction feature (ablation) | EERCF · MSRVTT-1K-Test t2v | Mean: 65.9 -> 67.8 | Helped |
| add text-driven video features (ablation) | EERCF · MSRVTT-1K-Test t2v | Mean: 65.2 -> 67.8 | Helped |
| add Pearson Constraint to L1-only features (ablation) | EERCF · MSRVTT-1K-Test t2v | Mean: 65.2 -> 65.7 | Helped |
| add Pearson Constraint to L1+L2+L3 features (ablation) | EERCF · MSRVTT-1K-Test t2v | Mean: 67.8 -> 68.7 | Helped |
| increase reranking top-k from 10 to 50 (ablation) | EERCF · MSRVTT-1K-Test t2v | Mean: 65.9 -> 68.7 | Helped |
| increase reranking top-k from 50 to 100 (ablation) | EERCF · MSRVTT-1K-Test t2v | Mean: 68.7 -> 67.9 | Hurt |
| use text-supervised EERCF frame-level reranking instead of X-CLIP reranking (ablation) | EERCF · MSRVTT-1K-Test t2v | Mean: 66.9 -> 67.8 | Helped |
| use text-supervised EERCF frame-level reranking instead of DRL reranking (ablation) | EERCF · MSRVTT-1K-Test t2v | Mean: 67.4 -> 67.8 | Helped |
| use text-supervised EERCF frame-level reranking instead of Video Sim (ablation) | EERCF · MSRVTT-1K-Test t2v | Mean: 63.2 -> 67.8 | Helped |
Surprising findings
- Using only coarse text-agnostic features for recall and reranking top-50 can outperform reranking all videos, with top-100 slightly decreasing performance, suggesting fine-grained features add noise at large candidate sets.
Limitations
- Fine-grained patch and frame features introduce noise for descriptive queries and large top-k reranking candidate sets.
- Some results (e.g., ViT-B/16 comparison with DRL*) rely on extra tricks (DSL or Q-Norm) and the main gains over DRL are comparable rather than superior.
Experimental setup
Datasets: MSR-VTT VATEX MSVD ActivityNet
Benchmarks: MSRVTT-1K-Test MSRVTT-3K-Test VATEX ActivityNet MSVD
Metrics: R@1 R@5 R@10 Mean recall FLOPs
Baselines: CLIP4Clip X-CLIP TS2-Net X-Pool DRL CLIP-VIP CenterCLIP TeachText LAFF BridgeFormer
Builds on: CLIP4Clip DRL X-CLIP CLIP
Closest papers by meaning
- TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval (ICLR 2025 · similarity 0.89)
- Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrieval (CVPR 2025 · similarity 0.88)
- Bridging the Semantic Granularity Gap Between Text and Frame Representations for Partially Relevant Video Retrieval (AAAI 2025 · similarity 0.88)
- From Play to Replay: Composed Video Retrieval for Temporally Fine-Grained Videos (NeurIPS 2025 · similarity 0.88)
- MUSE: Mamba Is Efficient Multi-scale Learner for Text-video Retrieval (AAAI 2025 · similarity 0.87)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.