Research map / Video / Video-language understanding
Event-Equalized Dense Video Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes E2DVC, a dense video captioning framework that uses non-trainable agglomerative clustering of visual features to form pseudo-events, which initialize decoder queries and enhance the encoder, to reduce temporal bias.
Problem
Dense video captioning models suffer from temporal bias: uneven event time distributions in training data and long-duration events dominating frame features cause short or out-of-distribution events to be missed or mislocalized.
Approach
A non-trainable Event Perception module clusters frame features with agglomerative hierarchical clustering and refines them temporally to form pseudo-events; these pseudo-event centers initialize event queries (Pseudo-Event Initialization) and pseudo-event labels are added to frame tokens in the encoder (Event-Enhanced Encoder), so all potential events get equal attention.
Key results
- On ActivityNet Captions with CLIP features, E2DVC reaches CIDEr 33.63 vs. PDVC* 29.97 and CM2 33.01.
- On YouCook2 with CLIP features, E2DVC reaches CIDEr 34.26 vs. PDVC* 29.69 (ablation baseline), and best among non-pretrained methods.
- Ablation on YouCook2 shows PEI and EEE each improve over baseline and combined yields the best F1 (28.64) and CIDEr (34.26).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PEI to initialize queries with pseudo-event locations (ablation) | E2DVC · YouCook2 | CIDEr: 31.15 -> 34.26 | Helped |
| Add EEE to incorporate pseudo-event labels during encoding (ablation) | E2DVC · YouCook2 | CIDEr: 30.92 -> 34.26 | Helped |
| Use E2DVC instead of CM2 with CLIP features | ActivityNet Captions · ActivityNet Captions | CIDEr: 33.01 -> 33.63 | Helped |
| Use E2DVC instead of CM2 with CLIP features | YouCook2 · YouCook2 | CIDEr: 31.66 -> 34.26 | Helped |
| Use E2DVC instead of CM2 with CLIP features | ActivityNet Captions · ActivityNet Captions | SODA_c: 6.18 -> 6.13 | Hurt |
| Vary the number of clusters Nc from 0 to 7 (ablation) | E2DVC · YouCook2 | CIDEr: 29.69; 31.45; 31.32; 32.57; 30.89; 34.26; 33.27; 30.10 | Mixed |
| Set the number of clusters Nc to 5 (ablation) | E2DVC · YouCook2 | CIDEr: 29.69 -> 34.26 | Helped |
| Vary temporal event threshold τ from 0 to 7 (ablation) | E2DVC · YouCook2 | CIDEr: 29.59; 31.69; 30.25; 31.76; 34.26; 31.50; 29.51; 31.13 | Mixed |
| Set the temporal event threshold τ to 4 (ablation) | E2DVC · YouCook2 | CIDEr: 29.69 -> 34.26 | Helped |
Surprising findings
- Vid2Seq, pretrained on about 1 million extra videos, outperforms E2DVC on YouCook2 captioning metrics.
- Localization of long-duration events remains imprecise; the authors argue captioning loss can make some localization deviations beneficial.
Limitations
- Performance is sensitive to hyperparameters such as cluster count Nc and event threshold tau.
- Long-duration event localization remains imprecise.
- Pretrained methods such as Vid2Seq outperform it on YouCook2.
Experimental setup
Datasets: ActivityNet Captions YouCook2
Benchmarks: ActivityNet Captions YouCook2
Metrics: BLEU4 METEOR CIDEr SODA_c event localization precision recall F1
Baselines: PDVC CM2 Vid2Seq MT ECHR E2ESG UEDVC OmniVID MFT
Builds on: PDVC DETR Deformable DETR Vid2Seq CM2
Closest papers by meaning
- Implicit Location-Caption Alignment via Complementary Masking for Weakly-Supervised Dense Video Captioning (AAAI 2025 · similarity 0.89)
- Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval (CVPR 2024 · similarity 0.89)
- Exploring Temporal Event Cues for Dense Video Captioning in Cyclic Co-Learning (AAAI 2025 · similarity 0.88)
- Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.88)
- DiffDVC: Accurate Event Detection for Dense Video Captioning via Diffusion Models (AAAI 2025 · similarity 0.88)
- DIBS: Enhancing Dense Video Captioning with Unlabeled Videos via Pseudo Boundary Enrichment and Online Refinement (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.