Research map / Video / Video-language understanding
Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a data-centric text enrichment framework for text-video retrieval that captions event-level clips during training and uses an LLM to generate diverse queries at test time, selected via a Farthest Query Sampling algorithm.
Problem
Text-video retrieval suffers from information asymmetry: videos are richer than their concise textual descriptions, so training data and user queries cover only fragments of video content.
Approach
Enrich text at training time by segmenting videos into event-level clips (KTS on CLIP features) and captioning middle frames with an image captioner, and at retrieval time expand the user query with LLM-generated diverse paraphrases, then select a relevant and diverse subset with Farthest Query Sampling (anchored on the user query) and aggregate per-query rankings by majority voting.
Key results
- On MSR-VTT 1K-A with CLIP ViT-B/32, X-Pool + Ours reaches 52.1 R@1 (vs 46.9 for X-Pool), and CLIP-VIP + Ours with DSL reaches 58.9 R@1.
- Oracle query selection raises MSR-VTT Rank-1 for the baseline from 46.7% to 61.4%.
- Using Phi-3.5 (3.8B) instead of GPT-4 for query enrichment gives 51.5 R@1 with FQS (k=2) versus 52.1 for GPT-4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add unified text enrichment to X-Pool | X-Pool, CLIP ViT-B/32 · MSR-VTT 1K-A | Rank@1: 46.7 -> 52.1 | Helped |
| Add unified text enrichment | X-Pool, CLIP ViT-B/32 · LSMDC t2v | R@1: 25.2 -> 28.3 | Helped |
| Add unified text enrichment | X-Pool, CLIP ViT-B/32 · MSVD t2v | R@1: 47.2 -> 51.1; 51.8 | Mixed |
| Add unified text enrichment | X-Pool, CLIP ViT-B/32 · VATEX t2v | R@1: 63.3; 66.6 | Mixed |
| Add training-phase event-aware caption enrichment (ablation) | X-Pool, CLIP ViT-B/32 · MSR-VTT | R@1: 46.7 -> 47.7 | Helped |
| Replace event-aware segmentation with fixed-time-interval captioning (ablation) | X-Pool, CLIP ViT-B/32 · MSR-VTT | R@1: 46.7 -> 47.0 | Helped |
| Add retrieval-phase enrichment only (ablation) | X-Pool, CLIP ViT-B/32 · MSR-VTT | R@1: 46.7 -> 50.2 | Helped |
| Use LLM query generation and majority voting instead of NLTK query generation and voting (ablation) | X-Pool, CLIP ViT-B/32 · MSR-VTT | R@1: 48.3 -> 49.6 | Helped |
| Concatenate generated LLM queries instead of using the baseline query (ablation) | X-Pool, CLIP ViT-B/32 · MSR-VTT | R@1: 46.7 -> 45.4 | Hurt |
| Average generated LLM text embeddings instead of using the baseline query (ablation) | X-Pool, CLIP ViT-B/32 · MSR-VTT | R@1: 46.7 -> 45.9 | Hurt |
| Select 2 enriched queries with FQS rather than 10 (ablation) | X-Pool, unified text enrichment · MSR-VTT | R@1: 49.6 -> 52.1 | Helped |
| Use FQS instead of k-DPP for query selection at k=2 (ablation) | X-Pool, unified text enrichment · MSR-VTT | R@1: 50.9 -> 52.1 | Helped |
Surprising findings
- Concatenating enriched queries or average-pooling text embeddings hurt retrieval below the baseline, while majority voting over per-query rankings helps.
- The Oracle query picks the original GT query 59.5% of the time, so the gap to oracle is partly from selection rather than enrichment alone.
Limitations
- Inference cost of the LLM and extra similarity computations (text-conditioned pooling) increases per-query latency (103 ms to 148 ms with query selection, excluding LLM).
- Enriched queries and captions can contain hallucinations from VLMs and LLMs, and the method does not address caption noise architecturally.
- Oracle selection requires ground truth and is not practical; a gap remains between the proposed selection and the oracle.
Experimental setup
Datasets: MSR-VTT MSVD LSMDC VATEX
Benchmarks: MSR-VTT MSVD LSMDC VATEX
Metrics: R@1 R@5 R@10 MdR MnR
Baselines: CLIP4Clip X-Pool CLIP-VIP T-MASS Cap4Video UATVR DSL k-DPP
Builds on: X-Pool CLIP CLIP4Clip kernel temporal segmentation
Closest papers by meaning
- Text Proxy: Decomposing Retrieval from a 1-to-N Relationship into N 1-to-1 Relationships for Text-Video Retrieval (AAAI 2025 · similarity 0.92)
- MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval (CVPR 2025 · similarity 0.88)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.88)
- Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrieval (CVPR 2025 · similarity 0.88)
- Learning Dynamic Similarity by Bidirectional Hierarchical Sliding Semantic Probe for Efficient Text Video Retrieval (AAAI 2025 · similarity 0.87)
- Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.