Research map / Video / Video-language understanding
HiCM²: Hierarchical Compact Memory Modeling for Dense Video Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HiCM², a dense video captioning model that retrieves from a hierarchical, LLM-summarized compact memory built by FINCH clustering of training captions, and reports state-of-the-art captioning results on YouCook2 and ViTT.
Problem
Dense video captioning requires localizing and describing events in untrimmed videos; prior retrieval-augmented methods use flat, redundant external memories of all training captions, which add noise.
Approach
Build a hierarchical external memory by bottom-up FINCH clustering of CLIP-encoded training captions, summarize each cluster with Llama3 70B into compact memory units, and retrieve top-down (abstract to detailed) per temporal anchor to condition a Vid2Seq-style T5 decoder.
Key results
- On YouCook2 validation, HiCM² reaches CIDEr 71.84 and SODA_c 10.73 versus Vid2Seq's 66.29 and 9.87 (reproduced).
- On ViTT test, HiCM² achieves CIDEr 51.29 versus Vid2Seq's 48.84.
- Ablation on YouCook2: LLM summarization over plain clustering improves CIDEr from 67.15 to 71.84; retrieving all training captions flat gives 67.90.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hierarchical compact memory retrieval with LLM summarization (ablation) | HiCM² · YouCook2 | CIDEr: 66.29 -> 71.84 | Helped |
| add hierarchical compact memory retrieval with LLM summarization | HiCM² · ViTT(test) | CIDEr: 48.84 -> 51.29 | Helped |
| add memory retrieval (ablation) | encoder-decoder without memory · YouCook2 | CIDEr: 66.29 -> 67.90 | Helped |
| add LLM summarization to hierarchical clustering (ablation) | Clustering memory · YouCook2 | CIDEr: 67.15 -> 71.84 | Helped |
| use all three hierarchical levels instead of any two-level combination (ablation) | HiCM² memory · YouCook2 | CIDEr: 69.81 -> 71.84 | Helped |
| retrieve 10 features per anchor at each level instead of 1 (ablation) | HiCM² · YouCook2 | CIDEr: 67.47 -> 71.84 | Helped |
| retrieve 20 features per anchor at each level instead of 10 (ablation) | HiCM² · YouCook2 | CIDEr: 71.84 -> 67.97 | Hurt |
| use Top-K memory selection instead of selecting only the maximum-similarity feature (ablation) | HiCM² · YouCook2 | CIDEr: 67.47 -> 71.84 | Helped |
| use 10 temporal anchors instead of 1 (ablation) | HiCM² retrieval · YouCook2 | CIDEr: 67.95 -> 71.84 | Helped |
| use 20 temporal anchors instead of 10 (ablation) | HiCM² retrieval · YouCook2 | CIDEr: 71.84 -> 66.37 | Hurt |
| concatenate retrieved features rather than pass them through a new cross-attention layer (ablation) | HiCM² decoder · YouCook2 | CIDEr: 69.90 -> 71.84 | Helped |
| enable hierarchical aggregation across memory levels (ablation) | HiCM² retrieval · YouCook2 | CIDEr: 69.67 -> 71.84 | Helped |
Surprising findings
- Localization gains are small: on ViTT, HiCM² F1 (45.98) is below Vid2Seq (46.21), and the paper notes that hierarchical aggregation without it enhances localization ability.
- Retrieving too many features per anchor (20) degrades performance, attributed to noise.
Limitations
- Gains over Vid2Seq on localization are marginal or negative on ViTT.
- Vid2Seq baseline results were reproduced by the authors, and ViTT used roughly 10-20% fewer videos due to YouTube availability.
- Memory is built only from training captions, so the approach depends on in-domain caption sources.
Experimental setup
Datasets: YouCook2 ViTT
Benchmarks: YouCook2 ViTT
Metrics: CIDEr METEOR SODA_c BLEU4 F1 precision recall
Baselines: PDVC CM² Streaming V2S DIBS Vid2Seq
Builds on: Vid2Seq CM² FINCH PDVC T5
Closest papers by meaning
- Streaming Dense Video Captioning (CVPR 2024 · similarity 0.90)
- Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval (CVPR 2024 · similarity 0.89)
- VidChain: Chain-of-Tasks with Metric-based Direct Preference Optimization for Dense Video Captioning (AAAI 2025 · similarity 0.88)
- DIBS: Enhancing Dense Video Captioning with Unlabeled Videos via Pseudo Boundary Enrichment and Online Refinement (CVPR 2024 · similarity 0.87)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.87)
- VideoLucy: Deep Memory Backtracking for Long Video Understanding (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.