Research map / Video / Video-language understanding
Streaming Dense Video Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a streaming dense video captioning model with a fixed-size K-means-style clustering memory for frame-by-frame input and a decoding-point algorithm that emits captions before the full video is seen, improving state of the art on ActivityNet, YouCook2 and ViTT.
Problem
Dense video captioning on long untrimmed videos is limited by heavy frame downsampling and by generating all event captions only after the full video has been processed, so it cannot stream outputs.
Approach
A fixed-size memory updated per frame by K-means-like clustering of incoming tokens with momentum weights, plus decoding points at intermediate timestamps where the decoder predicts all events finished so far, conditioned on earlier predicted captions as prefix.
Key results
- Streaming Vid2Seq reaches 37.8 CIDEr on ActivityNet vs 30.2 for Vid2Seq, and Streaming GIT reaches 41.2 CIDEr on ActivityNet, up to 11.0 points over prior visual-only SOTA.
- Streaming GIT improves YouCook2 CIDEr to 15.4 vs 12.1 for GIT, while Streaming Vid2Seq reaches 32.9 vs 25.3 for Vid2Seq.
- On paragraph captioning, Streaming GIT achieves 33.4 CIDEr on ActivityNet and 33.9 on YouCook2, above GIT's 32.5 and 28.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add streaming dense captioning to GIT | GIT · ActivityNet validation | CIDEr: 29.8 -> 41.2 | Helped |
| Add streaming dense captioning to Vid2Seq | Vid2Seq · YouCook2 validation | CIDEr: 25.3 -> 32.9 | Helped |
| Use streaming input memory | GIT paragraph captioning · ActivityNet validation paragraph captioning | CIDEr: 32.5 -> 33.4 | Helped |
| Replace clustering memory with temporal pooling (ablation) | GIT · ActivityNet, T=64 | CIDEr: 30.6 -> 26.8 | Hurt |
| Replace clustering memory with EMA (ablation) | GIT · ActivityNet, T=64 | CIDEr: 30.6 -> 22.0 | Hurt |
| Replace clustering memory with MovieChat memory (ablation) | GIT · ActivityNet, T=64 | CIDEr: 28.9 -> 30.6 | Hurt |
| Use the momentum term in clustering (ablation) | Clustering memory · ActivityNet, GIT, 64 frames | CIDEr: 29.7 -> 30.6 | Helped |
| Increase decoding points during training from 1 to 16 (ablation) | GIT · ActivityNet | CIDEr: 30.6 -> 40.6 | Helped |
| Provide previous captions as context after the first decoding point (ablation) | GIT streaming decoder · ActivityNet | CIDEr: 23.1 -> 40.6 | Helped |
| Add timestamps to the previous-caption prefix (ablation) | GIT streaming decoder · ActivityNet | CIDEr: 40.6 -> 39.3 | Hurt |
| Use random prefix masking augmentation during training (ablation) | GIT streaming decoder · ActivityNet | CIDEr: 37.6 -> 40.6 | Helped |
| Increase memory tokens from 257 to 257 × 2 (ablation) | GIT clustering memory · ActivityNet, 64 frames | CIDEr: 29.4 -> 30.6 | Helped |
Surprising findings
- Without past-caption prefix at decoding points, the streaming model underperforms the non-streaming baseline (23.1 vs 30.6 CIDEr), due to duplicated predictions.
- Naive spatial or temporal pooling of tokens gets worse as more frames are used, while the clustering memory keeps improving.
Limitations
- Using many decoding points at inference can still produce duplicate captions, and NMS is left as future work.
- Evaluated only on visual inputs; ASR is not used and would improve results.
- Existing benchmarks do not require reasoning over very long videos.
Experimental setup
Datasets: ActivityNet Captions YouCook2 ViTT WebLI YT-Temporal
Benchmarks: ActivityNet Captions YouCook2 ViTT
Metrics: CIDEr SODA METEOR F1
Baselines: MT E2ESG PDVC GIT Vid2Seq MovieChat TeSTra MFT
Builds on: GIT Vid2Seq TeSTra MovieChat PDVC
Closest papers by meaning
- HiCM²: Hierarchical Compact Memory Modeling for Dense Video Captioning (AAAI 2025 · similarity 0.90)
- Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.88)
- Fine-Grained Captioning of Long Videos through Scene Graph Consolidation (ICML 2025 · similarity 0.88)
- Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval (CVPR 2024 · similarity 0.87)
- Event-Equalized Dense Video Captioning (CVPR 2025 · similarity 0.87)
- DIBS: Enhancing Dense Video Captioning with Unlabeled Videos via Pseudo Boundary Enrichment and Online Refinement (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.