Research map / Video / Video-language understanding
From Play to Replay: Composed Video Retrieval for Temporally Fine-Grained Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TF-CoVR, a 180K-triplet benchmark for temporally fine-grained composed video retrieval on gymnastics and diving, and TF-CoVR-Base, a two-stage AIM pretraining plus contrastive composed-query model that sets new results.
Problem
Existing composed video retrieval benchmarks emphasize appearance or coarse event changes and offer a single target per query, so they do not test subtle, fast-paced temporal differences.
Approach
Builds triplets from FineGym and FineDiving sub-action labels by pairing CLIP-similar labels and using GPT-4o to write modification texts, giving each query multiple valid targets; a two-stage model pretrains a video encoder (AIM) on fine-grained action classification, then fuses the query video and text for contrastive retrieval.
Key results
- Zero-shot TF-CoVR-Base (AIM, 32 frames) reaches 7.51 mAP@50 versus 5.92 for LanguageBind and 5.22 for E5-V.
- Fine-tuned TF-CoVR-Base raises mAP@50 from 19.83 (BLIP-CoVR-ECDE) to 27.22.
- Using HN-NCE with positive hard-negative weighting underperforms plain InfoNCE; weight 0 gives 27.22 vs 25.37 at weight 0.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TF-CoVR-Base rather than LanguageBind in zero-shot retrieval | TF-CoVR · TF-CoVR | mAP@50: 5.92 -> 7.51 | Helped |
| Fine-tune TF-CoVR-Base on TF-CoVR | TF-CoVR-Base · TF-CoVR | mAP@50: 19.83 -> 27.22 | Helped |
| Add Stage-1 temporal pretraining (ablation) | TF-CoVR-Base · TF-CoVR | mAP@50: 22.61 -> 27.22 | Helped |
| Set hard-negative weighting to 0.0 (InfoNCE) rather than 0.7 (ablation) | TF-CoVR-Base · TF-CoVR | mAP@50: 25.37 -> 27.22 | Helped |
| Set hard-negative weighting to 0.0 (InfoNCE) rather than 0.5 (ablation) | TF-CoVR-Base · TF-CoVR | mAP@50: 25.91 -> 27.22 | Helped |
| Set hard-negative weighting to 0.0 (InfoNCE) rather than 0.3 (ablation) | TF-CoVR-Base · TF-CoVR | mAP@50: 26.16 -> 27.22 | Helped |
| Use BLIP instead of CLIP as the text encoder (ablation) | TF-CoVR-Base · TF-CoVR | mAP@50: 23.64 ± 0.27 -> 25.88 ± 0.25 | Helped |
Surprising findings
- Hard-negative weighting in HN-NCE hurts performance relative to standard InfoNCE on this fine-grained benchmark.
- Zero-shot general multimodal embedding models (E5-V) outperform most specialized CoVR methods trained on WebVid-CoVR.
Limitations
- Requires expert temporal annotations from FineGym and FineDiving, which are costly to scale.
- TF-CoVR-Base is two-stage rather than end-to-end.
- Domain coverage limited to gymnastics and diving.
Experimental setup
Datasets: TF-CoVR FineGym FineDiving WebVid-CoVR
Benchmarks: TF-CoVR
Metrics: mAP@5 mAP@10 mAP@25 mAP@50
Baselines: LanguageBind E5-V MM-Embed GME-Qwen2-VL-2B BLIP-CoVR BLIP-CoVR-ECDE TF-CVR CLIP BLIP2
Builds on: AIM BLIP-CoVR CIRCO mAP@K InfoNCE HN-NCE
Closest papers by meaning
- Composed Video Retrieval via Enriched Context and Discriminative Embeddings (CVPR 2024 · similarity 0.92)
- CoVR: Learning Composed Video Retrieval from Web Video Captions (AAAI 2024 · similarity 0.91)
- MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence (NeurIPS 2025 · similarity 0.89)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.88)
- Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learning (AAAI 2024 · similarity 0.88)
- TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.