Research map / Video / Video-language understanding
CoVR: Learning Composed Video Retrieval from Web Video Captions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes automatic generation of composed video retrieval triplets from web video-caption pairs (WebVid-CoVR, 1.6M triplets) and a BLIP-based CoVR model, showing zero-shot transfer to composed image retrieval benchmarks.
Problem
Composed video retrieval (CoVR) lacks training data: manually annotating video-text-video triplets is expensive, and composed image retrieval (CoIR) datasets are small.
Approach
Mine video pairs whose captions differ by a single word from a web video-caption corpus, use an LLM finetuned on InstructPix2Pix edit annotations to generate the modification text between captions, and train a BLIP-based composed retrieval model with HN-NCE contrastive loss on the resulting triplets.
Key results
- Zero-shot CIRR R@1 of 38.48 for CoVR-BLIP trained on WebVid-CoVR, versus 30.89 for CASE (LaSCo) and 19.37 for CompoDiff.
- On WebVid-CoVR-Test, the finetuned model with cross-attention fusion and 15 frames reaches R@1 53.13 versus 19.68 for text-only BLIP.
- Zero-shot FashionIQ R@10 of 27.70 for CoVR-BLIP trained on WebVid-CoVR, versus 24.70 for Pic2Word.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on WebVid-CoVR instead of using frozen pretrained embeddings | CoVR-BLIP · WebVid-CoVR-Test | R@1: 45.46 -> 53.13 | Helped |
| Combine visual and text query modalities (ablation) | CoVR-BLIP · WebVid-CoVR-Test | R@1: 38.89 -> 53.13 | Helped |
| Use cross-attention instead of MLP fusion (ablation) | CoVR-BLIP · WebVid-CoVR-Test | R@1: 50.63 -> 53.13 | Helped |
| Use 15 target-video frames instead of the middle frame (ablation) | CoVR-BLIP · WebVid-CoVR-Test | R@1: 51.80 -> 53.13 | Helped |
| Train on WebVid-CoVR for zero-shot transfer | CoVR-BLIP · CIRR | R@1: 19.76 -> 38.48 | Helped |
| Train on WebVid-CoVR for zero-shot transfer | CoVR-BLIP · FashionIQ | R@10: 16.00 -> 27.70 | Helped |
| Increase the initial video-caption seed set from 0 to 2.5M videos (ablation) | CoVR-BLIP · CIRR | MeanR: 45.88 -> 68.48 | Helped |
| Remove filtering from the generated training data (ablation) | CoVR-BLIP · CIRR | MeanR: 68.48 -> 67.41 | Hurt |
| Use MTG-LLM instead of rule-based modification text generation (ablation) | CoVR-BLIP · CIRR | MeanR: 46.16 -> 68.48 | Helped |
| Use MTG-LLM instead of paraphrased rule-based text (ablation) | CoVR-BLIP · CIRR | MeanR: 64.33 -> 68.48 | Helped |
| Iterate over distinct target videos instead of triplets (ablation) | CoVR-BLIP · CIRR | MeanR: 65.59 -> 68.48 | Helped |
| Use HN-NCE hard-negative upsampling (ablation) | CoVR-BLIP · CIRR | MeanR: 66.07 -> 68.48 | Helped |
Surprising findings
- Training on video-derived triplets transfers to image composed retrieval and beats models trained on larger synthetic image data (CompoDiff ST18M) in zero-shot CIRR.
- Paraphrasing rule-based modification texts with GPT-3.5 substantially improves over raw rule-based texts, though still below the MTG-LLM.
Limitations
- Generated modification text is based only on captions and ignores visual content, so it may describe changes not visible in the videos.
- Only single-word caption differences are mined, so multi-change modifications are not covered.
- About 22% of automatically generated test candidates were noisy and discarded during manual annotation.
Experimental setup
Datasets: WebVid2M WebVid10M InstructPix2Pix editing dataset CIRR FashionIQ COCO
Benchmarks: WebVid-CoVR-Test CIRR FashionIQ
Metrics: recall@1 recall@5 recall@10 recall@50 MeanR
Baselines: Combiner CASE CompoDiff Pic2Word SEARLE CLIP BLIP
Builds on: BLIP HN-NCE InstructPix2Pix LLaMA Combiner
Closest papers by meaning
- Composed Video Retrieval via Enriched Context and Discriminative Embeddings (CVPR 2024 · similarity 0.94)
- From Play to Replay: Composed Video Retrieval for Temporally Fine-Grained Videos (NeurIPS 2025 · similarity 0.91)
- Data Roaming and Quality Assessment for Composed Image Retrieval (AAAI 2024 · similarity 0.91)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.89)
- CoLLM: A Large Language Model for Composed Image Retrieval (CVPR 2025 · similarity 0.88)
- MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.