Research map / Video / Video-language understanding
Text Proxy: Decomposing Retrieval from a 1-to-N Relationship into N 1-to-1 Relationships for Text-Video Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TV-ProxyNet, which replaces each text query with learned text proxies (steered by a director and dash) to turn 1-to-N text-video matching into N 1-to-1 proxy-video matches, built on CLIP-ViP.
Problem
Text-video retrieval is framed as a 1-to-N matching problem that suffers from irregular data (overly simple captions, videos with changing content), causing errors in matching.
Approach
Generate a per-pair text proxy from the text query via a learned direction leader (cross-attention over video proxy tokens), a director vector, and a dash scalar controlling distance; train with proxy-video InfoNCE losses and combine proxy-video and text-video logits at inference.
Key results
- On MSRVTT with ViT-B/32, TV-ProxyNet reaches 52.3 R@1 vs 50.1 for CLIP-ViP (+2.2).
- On ActivityNet Captions with ViT-B/16, R@1 is 56.7 vs 53.4 for CLIP-ViP (+3.3).
- On DiDeMo with ViT-B/32, R@1 is 50.6 vs 48.6 for CLIP-ViP, but below T-MASS (50.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TV-ProxyNet instead of CLIP-ViP | CLIP-ViT-B/32 · MSRVTT | R@1: 52.3% v.s. 50.1% | Helped |
| Use TV-ProxyNet instead of Cap4Video | CLIP-ViT-B/32 · DiDeMo | R@1: 50.6 v.s. 52.0 | Hurt |
| Use TV-ProxyNet instead of CLIP-ViP | CLIP-ViT-B/16 · MSRVTT | R@1 | Helped |
| Use TV-ProxyNet instead of T-MASS | CLIP-ViT-B/16 · DiDeMo | R@1 | Hurt |
| Use TV-ProxyNet instead of CLIP-ViP | CLIP-ViT-B/32 · ActivityNet Captions | R@1: 1.9% (53.0% v.s. 51.1%) | Helped |
| Use TV-ProxyNet instead of CLIP-ViP | CLIP-ViT-B/16 · ActivityNet Captions | R@1: 3.3% (56.7% v.s. 53.4%) | Helped |
| Add L_p to the baseline loss (ablation) | TV-ProxyNet with L_r only · MSRVTT-9k | R@1: 50.1 -> 52.0 | Helped |
| Add L_pos without L_p (ablation) | TV-ProxyNet with L_r only · MSRVTT-9k | R@1: 50.1 -> 50.6 | No clear effect |
| Add L_pos alongside L_p (ablation) | TV-ProxyNet with L_r + L_p · MSRVTT-9k | R@1: 52.0 -> 52.3 | Helped |
| Use mean-similarity dash instead of no dash (ablation) | TV-ProxyNet · MSRVTT-9k | R@1: 51.9 -> 52.3 | Helped |
| Increase mean-similarity dash iterations from 1 to 3 (ablation) | TV-ProxyNet · MSRVTT-9k | R@1: 51.9 -> 52.5 | Helped |
| Increase mean-similarity dash iterations from 3 to 5 (ablation) | TV-ProxyNet · MSRVTT-9k | R@1: 52.5 -> 52.3 | Hurt |
Surprising findings
- Using the proxy alone without the dash mechanism gives weaker and non-scalable results with more iterations, while the dash is central to performance.
- Adding L_pos alone with proxy-video logit fusion gives minimal gains; performance peaks at gamma=0 in that setting.
Limitations
- Does not resolve ambiguous text correspondences in videos (failure case with repeated content).
- Does not outperform T-MASS on DiDeMo ViT-B/16 and ViT-B/32 results are mixed; gains depend on hyperparameters delta/eta and dataset-specific dash choice.
- Space complexity grows as O(B^2) due to generating B proxies per text input.
Experimental setup
Datasets: MSRVTT DiDeMo ActivityNet Captions
Benchmarks: MSRVTT DiDeMo ActivityNet Captions
Metrics: R@1 R@5 R@10 MdR MnR
Baselines: CLIP-ViP T-MASS Cap4Video HBI ProST UATVR X-Pool TS2-Net CLIP4Clip CenterCLIP DRL CAMoE
Builds on: CLIP-ViP CLIP4Clip InfoNCE
Closest papers by meaning
- Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach (ICLR 2025 · similarity 0.92)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.88)
- TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval (ICLR 2025 · similarity 0.88)
- Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrieval (CVPR 2025 · similarity 0.87)
- Learning Dynamic Similarity by Bidirectional Hierarchical Sliding Semantic Probe for Efficient Text Video Retrieval (AAAI 2025 · similarity 0.87)
- DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.