Research map / Video / Video-language understanding
Composed Video Retrieval via Enriched Context and Discriminative Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a composed video retrieval framework that adds machine-generated detailed frame descriptions and fuses vision, text and vision-text embeddings with hard-negative contrastive losses, built on CoVR-BLIP.
Problem
Composed video retrieval (CoVR) retrieves target videos from a reference video plus change text, but the baseline's fused representation is biased toward the change text and loses query-specific context, and the target is represented only by a visual embedding.
Approach
Augment the query video with detailed language descriptions generated by a multimodal conversation model, sum three grounded multimodal embeddings (q+t, q+d, e(d)+t), and align them against three target databases (visual, multimodal, text-only) using hard-negative contrastive losses.
Key results
- On WebVid-CoVR test, the method reaches R@1 60.12 vs. 53.13 for CoVR-BLIP, a gain of about 7 points.
- In zero-shot CIRR, R@1 reaches 40.12 vs. 38.48 for CoVR-BLIP (trained on WebVid-CoVR).
- Using detailed descriptions instead of WebVid short captions improves R@1 from 58.23 to 60.12 on WebVid-CoVR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use video, detailed description, and change text together (ablation) | Our Approach · WebVid-CoVR test set | R@1: 56.26 -> 60.12 | Helped |
| Add the reference video to description-only input (ablation) | Our Approach · WebVid-CoVR test set | R@1: 39.92 -> 42.19 | Helped |
| Add description to video-only input (ablation) | Our Approach · WebVid-CoVR test set | R@1: 40.51 -> 42.19 | Helped |
| Add change text to video and description (ablation) | Our Approach · WebVid-CoVR test set | R@1: 42.19 -> 60.12 | Helped |
| Use all three target losses (Lve, Lmme, Lte) rather than Lve and Lmme (ablation) | Our Approach · WebVid-CoVR test set | R@1: 59.12 -> 60.12 | Helped |
| Add visual-embedding loss Lve to Lmme and Lte (ablation) | Our Approach · WebVid-CoVR test set | R@1: 58.94 -> 60.12 | Helped |
| Add multimodal-embedding loss Lmme to Lve and Lte (ablation) | Our Approach · WebVid-CoVR test set | R@1: 58.94 -> 60.12 | Helped |
| Add text-embedding loss Lte to Lve and Lmme (ablation) | Our Approach · WebVid-CoVR test set | R@1: 59.12 -> 60.12 | Helped |
| Replace short WebVid captions with detailed descriptions (ablation) | Our Approach · WebVid-CoVR test set | R@1: 58.23 -> 60.12 | Helped |
| Use our method instead of CoVR-BLIP with the same training setup | WebVid-CoVR retrieval · WebVid-CoVR test set | R@1: 53.13 -> 60.12 | Helped |
| Use our method instead of CoVR-BLIP | Zero-shot CIRR retrieval · CIRR test set | R@1: 38.48 -> 40.12 | Helped |
| Use our method instead of CoVR-BLIP | Zero-shot FashionIQ retrieval · FashionIQ val, Dress | R@10: 21.95 -> 24.57 | Helped |
Surprising findings
- Using only the change text as input yields much lower recall than using descriptions or video alone, and the model acts as plain retrieval without it.
Limitations
- Generated descriptions can hallucinate; a similarity threshold is used to filter them, and no human verification is described.
- Training relies on a frozen BLIP/BLIP-2 backbone and the synthetic WebVid-CoVR triplets.
Experimental setup
Datasets: WebVid-CoVR CIRR FashionIQ WebVid captions
Benchmarks: WebVid-CoVR CIRR FashionIQ
Metrics: Recall@1 Recall@5 Recall@10 Recall@50
Baselines: CoVR-BLIP Pic2Word CompoDiff CASE Combiner TIRG
Builds on: CoVR-BLIP BLIP-2 hard-negative NCE
Closest papers by meaning
- CoVR: Learning Composed Video Retrieval from Web Video Captions (AAAI 2024 · similarity 0.94)
- From Play to Replay: Composed Video Retrieval for Temporally Fine-Grained Videos (NeurIPS 2025 · similarity 0.92)
- Data Roaming and Quality Assessment for Composed Image Retrieval (AAAI 2024 · similarity 0.89)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.88)
- MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence (NeurIPS 2025 · similarity 0.87)
- Bridging the Semantic Granularity Gap Between Text and Frame Representations for Partially Relevant Video Retrieval (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.