Research map / Video / Video-language understanding
Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CM2, a dense video captioning model that retrieves sentence-level text memories from an external CLIP-indexed caption bank via video-to-text matching and fuses them with visual features through versatile encoder-decoder cross-attention.
Problem
Dense video captioning (localizing and captioning all events in untrimmed video) is hard using only visual input without semantic content or large-scale video pretraining.
Approach
Retrieve relevant caption sentences from an external memory bank built from in-domain training captions using CLIP segment-level video-to-text similarity, aggregate them per temporal anchor, and inject them through a weight-shared versatile encoder and separate visual/textual cross-attention in a deformable transformer decoder.
Key results
- On ActivityNet Captions, CM2 reaches CIDEr 33.01 and METEOR 8.55 versus 29.97 and 8.06 for PDVC under the same CLIP backbone.
- On YouCook2, CM2 reaches CIDEr 31.66 versus 29.69 for PDVC, while Vid2Seq (1M pretraining videos) reaches 47.10.
- Oracle retrieval with ground-truth proposals reaches CIDEr 183.95 on YouCook2, showing large headroom for memory retrieval.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cross-modal memory retrieval (ablation) | CM2 · ActivityNet Captions | CIDEr: 31.24 -> 33.01 | Helped |
| add cross-modal memory retrieval (ablation) | CM2 · YouCook2 | CIDEr: 23.67 -> 31.66 | Helped |
| use ground-truth caption queries for oracle retrieval (ablation) | CM2 on YouCook2 · YouCook2 | CIDEr: 31.66 -> 53.55 | Helped |
| use ground-truth event proposals with oracle retrieval (ablation) | CM2 on YouCook2 · YouCook2 | CIDEr: 53.55 -> 183.95 | Helped |
| add separate visual and textual encoding (ablation) | CM2 · YouCook2 | CIDEr: 29.49 -> 30.86 | Helped |
| add textual cross-attention (ablation) | CM2 · YouCook2 | CIDEr: 29.49 -> 28.40; 30.86 -> 30.94 | Mixed |
| add weight sharing between modality encoders (ablation) | CM2 · YouCook2 | CIDEr: 30.94 -> 31.66 | Helped |
| use 10 retrieval anchors instead of 1 (ablation) | CM2 · YouCook2 | CIDEr: 27.97 -> 31.36 | Helped |
| increase retrieval anchors from 10 to 30 (ablation) | CM2 · YouCook2 | CIDEr: 31.36 -> 28.41 | Hurt |
| increase retrieval anchors from 30 to 50 (ablation) | CM2 · YouCook2 | CIDEr: 28.41 -> 31.66 | Helped |
| increase retrieval anchors from 50 to 70 (ablation) | CM2 · YouCook2 | CIDEr: 31.66 -> 32.73 | Helped |
| increase retrieval anchors from 70 to 90 (ablation) | CM2 · YouCook2 | CIDEr: 32.73 -> 29.88 | Hurt |
Surprising findings
- Oracle retrieval using ground-truth captions as queries yields very large gains (YouCook2 CIDEr 183.95), far above the practical retrieval method (31.66).
Limitations
- Video-to-text matching projects video features into the image-text CLIP space by averaging frames, potentially losing temporal information.
- Too many retrieved features introduce noise and degrade performance.
- YouCook2 results lag Vid2Seq, which uses 1M pretraining videos.
Experimental setup
Datasets: ActivityNet Captions YouCook2
Benchmarks: ActivityNet Captions YouCook2
Metrics: CIDEr METEOR BLEU4 SODA_c precision recall F1
Baselines: PDVC Vid2Seq MT ECHR UEDVC E2ESG
Builds on: PDVC CLIP deformable transformer retrieval-augmented generation
Closest papers by meaning
- HiCM²: Hierarchical Compact Memory Modeling for Dense Video Captioning (AAAI 2025 · similarity 0.89)
- Event-Equalized Dense Video Captioning (CVPR 2025 · similarity 0.89)
- Implicit Location-Caption Alignment via Complementary Masking for Weakly-Supervised Dense Video Captioning (AAAI 2025 · similarity 0.87)
- Exploring Temporal Event Cues for Dense Video Captioning in Cyclic Co-Learning (AAAI 2025 · similarity 0.87)
- Streaming Dense Video Captioning (CVPR 2024 · similarity 0.87)
- Exploiting Auxiliary Caption for Video Grounding (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.