Research map / Video / Video-language understanding
Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NarVid, which generates per-frame captions (narration) with an LMM and uses them through cross-modal co-attention, query-aware nucleus filtering, query-narration matching, and a cross-view hard negative loss for text-video retrieval.
Problem
Text-video retrieval struggles with the cross-modality gap and time-varying content; existing caption-based methods use a single video-level caption that misses temporal changes and can propagate generation errors.
Approach
Replace video-level captions with frame-level captions (narration) used across the whole matching pipeline: co-attention between frames and captions, adaptive query-aware nucleus filtering, a dual query-video/query-narration similarity, and a hard-negative loss that combines both views.
Key results
- NarVid reaches 52.7% R@1 on MSR-VTT with ViT-B/16, matching T-MASS on R@1 and exceeding Cap4Video (51.4%).
- NarVid achieves 53.1% R@1 on MSVD, 68.4% on VATEX, and 53.4% on DiDeMo (text-to-video).
- Ablation on MSR-VTT shows narration matching raises R@1 from 44.5 to 48.6 with ViT-B/32; full model reaches 51.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add narration matching (ablation) | NarVid · MSR-VTT 1k-A | R@1: 44.5 -> 48.6 | Helped |
| Add cross-modal interaction (ablation) | NarVid with narration matching · MSR-VTT 1k-A | R@1: 48.6 -> 49.5 | Helped |
| Add query-aware filtering (ablation) | NarVid with cross-modal interaction · MSR-VTT 1k-A | R@1: 49.5 -> 50.4 | Helped |
| Add cross-view hard negative loss (ablation) | NarVid with query-aware filtering · MSR-VTT 1k-A | R@1: 50.4 -> 51.0 | Helped |
| Add query-aware filtering instead of using no filtering (ablation) | NarVid · MSR-VTT 1k-A | R@1: 48.1 -> 51.0 | Helped |
| Replace fixed top-k filtering with adaptive nucleus filtering (ablation) | NarVid · MSR-VTT 1k-A | R@1: 50.3 -> 51.0 | Helped |
| Set hard negative loss weight α to 1.0 rather than 0 (ablation) | NarVid · MSR-VTT 1k-A | R@1: 50.4 -> 51.0 | Helped |
| Increase hard negative loss weight α from 1.0 to 1.5 (ablation) | NarVid · MSR-VTT 1k-A | R@1: 51.0 -> 50.1 | Hurt |
| Use NarVid instead of T-MASS | CLIP ViT-B/16 · MSR-VTT 1k-A, text-to-video | R@1: 52.7 -> 52.7 | No clear effect |
| Use NarVid instead of Cap4Video | CLIP ViT-B/16 · MSVD, text-to-video | R@1: 51.8 -> 53.1 | Helped |
| Use NarVid instead of Cap4Video | CLIP ViT-B/16 · VATEX, text-to-video | R@1: 66.6 -> 68.4 | Helped |
| Use NarVid instead of T-MASS | CLIP ViT-B/16 · DiDeMo, text-to-video | R@1: 53.3 -> 53.4 | Helped |
Surprising findings
- More powerful VLM captioners produce detailed captions with many query-irrelevant words that can hurt retrieval.
- Simply adding frame-level captions via query-narration matching is slightly below Cap4Video, showing that naive use is insufficient.
Limitations
- Narration generation requires precomputation and storage for each video.
- Retrieval gains depend heavily on caption generator quality.
- Lower R@5 and R@10 than T-MASS on several benchmarks.
Experimental setup
Datasets: MSR-VTT MSVD VATEX DiDeMo
Benchmarks: MSR-VTT MSVD VATEX DiDeMo
Metrics: R@1 R@5 R@10 MdR MnR
Baselines: CLIP4Clip X-CLIP X-Pool UATVR TEFAL PAU Cap4Video MPT ProTA T-MASS
Builds on: CLIP4Clip Cap4Video InfoNCE nucleus sampling
Closest papers by meaning
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.88)
- DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval (CVPR 2025 · similarity 0.87)
- Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach (ICLR 2025 · similarity 0.87)
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · similarity 0.87)
- Learning Audio-guided Video Representation with Gated Attention for Video-Text Retrieval (CVPR 2025 · similarity 0.87)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.