Research map / Video / Video-language understanding
Exploring Temporal Event Cues for Dense Video Captioning in Cyclic Co-Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MCCL, a dense video captioning network that retrieves sentences via CLIP video-to-text retrieval, detects frame-level concepts with weakly supervised multiple instance learning, and uses cyclic co-learning between generator and localizer.
Problem
Dense video captioning requires detecting and describing all events in untrimmed videos, but existing methods lack effective interaction between event semantic perception and temporal localization and use limited event cues.
Approach
Combine retrieval-enhanced visual features, weakly supervised frame-level concept detection (MIL with concept triplet contrastive loss) as temporal event cues, and a cyclic loss where the generator guides localization via semantic matching and the localizer guides semantics via location matching, built on PDVC.
Key results
- On ActivityNet Captions with ground-truth proposals, MCCL reaches 63.01 CIDEr, a 4.04% improvement over the prior best.
- On YouCook2 with learned proposals, MCCL reaches 36.09 CIDEr, a 5.94% improvement over CM^2 (per text), with no extra pretraining.
- On ActivityNet localization, the cyclic loss yields the best F1 of 55.23 versus 54.91 for CM^2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MCCL instead of CM² | CLIP-based dense video captioning · ActivityNet Captions, learned proposals | CIDEr: 32.89 -> 34.92 | Helped |
| use MCCL instead of CM² | CLIP-based dense video captioning · YouCook2, learned proposals | CIDEr: 30.15 -> 36.09 | Helped |
| add video-to-text retrieval (ablation) | MCCL · ActivityNet Captions, learned proposals | CIDEr: 27.74 -> 31.32 | Helped |
| add multiple concept detection alongside video-to-text retrieval (ablation) | MCCL · ActivityNet Captions, learned proposals | CIDEr: 31.32 -> 32.97 | Helped |
| add cyclic co-learning to video-to-text retrieval and multiple concept detection (ablation) | MCCL · ActivityNet Captions, learned proposals | CIDEr: 32.97 -> 34.92 | Helped |
| add concept features to video features (ablation) | MCCL · ActivityNet Captions, learned proposals | CIDEr: 32.46 -> 34.92 | Helped |
| use cyclic loss instead of semantic-guided loss (ablation) | MCCL · ActivityNet Captions, learned proposals | F1: 52.79 -> 55.23 | Helped |
| use cyclic loss instead of localization-guided loss (ablation) | MCCL · ActivityNet Captions, learned proposals | F1: 54.28 -> 55.23 | Helped |
| add concept contrastive learning with N_S=10 rather than no contrastive learning (ablation) | MCCL · ActivityNet Captions, learned proposals | CIDEr: 30.11 -> 34.92 | Helped |
| increase retrieved sentence count from N_K=10 to N_K=30 (ablation) | MCCL · ActivityNet Captions, learned proposals | SODA_c: 6.16 -> 6.34 | Helped |
| increase retrieved sentence count from N_K=10 to N_K=30 (ablation) | MCCL · ActivityNet Captions, learned proposals | CIDEr: 34.92 -> 34.90 | Hurt |
| vary the number of concepts from N_C=500 to N_C=600 (ablation) | MCCL · ActivityNet Captions, learned proposals | BLEU4: 2.68 -> 2.75 | Helped |
Surprising findings
- Frame-level concept detection alone gives limited standalone gains, but combined with video-to-text retrieval it significantly improves all metrics.
- The semantic-guided loss alone achieves highest precision but lower recall and weaker captions than the localization-guided loss.
Limitations
- Relationships between objects are hard to discern from visual features alone.
- Concept vocabulary construction affects caption quality; irrelevant or redundant concepts can mislead detection.
- Inconsistent concept naming (e.g., 'boy' and 'man' for the same person) can cause inconsistencies across event descriptions.
Experimental setup
Datasets: ActivityNet Captions YouCook2
Benchmarks: ActivityNet Captions YouCook2
Metrics: CIDEr BLEU4 METEOR SODA_c recall precision F1
Baselines: HRNN DCE DVC GPaS E2E-MT PDVC MDVC BMT GS-MS-FTN Vid2Seq DIBS CM^2
Builds on: PDVC CM^2 Hungarian matching deformable transformer
Closest papers by meaning
- Event-Equalized Dense Video Captioning (CVPR 2025 · similarity 0.88)
- Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval (CVPR 2024 · similarity 0.87)
- Implicit Location-Caption Alignment via Complementary Masking for Weakly-Supervised Dense Video Captioning (AAAI 2025 · similarity 0.87)
- Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.85)
- Comprehensive Visual Grounding for Video Description (AAAI 2024 · similarity 0.85)
- Learning Multi-Scale Video-Text Correspondence for Weakly Supervised Temporal Article Gronding (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.