Research map / Video / Video segmentation and tracking
DiffDVC: Accurate Event Detection for Dense Video Captioning via Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffDVC, a diffusion-based dense video captioning model whose boundary-sensitive detector attends to frames around noisy proposal boundaries with adaptive sampling, improving event detection and captioning on ActivityNet and YouCook2.
Problem
Diffusion-based event detectors borrowed from image object detection ignore that video event classification relies on the middle of proposals rather than boundaries, so the classifier is insensitive to boundary noise, hurting dense video captioning.
Approach
A boundary-sensitive detector that uses noisy proposal start/end times as reference points and learns deformable sampling offsets and attention weights around them, so the classification head becomes sensitive to boundary changes within a dynamically sized range.
Key results
- On ActivityNet Captions with C3D features, DiffDVC reports CIDEr 28.20 vs PDVC 25.87 and SODA_c 5.71 vs 5.26.
- On ActivityNet-1.3 event detection, DiffDVC achieves 35.8 average mAP, above DiffTAD (34.8) but below TriDet (36.8).
- On YouCook2 event detection, DiffDVC reaches F1 31.22 vs PDVC 28.29 (TSN features).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add boundary-sensitive detector (vs vanilla detector) (ablation) | DiffDVC · ActivityNet Captions (C3D/TSN/I3D) | F1: +1.15 (C3D), +1.48 (TSN), +1.05 (I3D) | Helped |
| add boundary-sensitive detector (vs vanilla detector) (ablation) | DiffDVC · ActivityNet Captions (C3D) | CIDEr: +1.36 | Helped |
| add boundary-sensitive detector (vs vanilla detector) (ablation) | DiffDVC · YouCook2 | CIDEr: +4.69 | Helped |
| add boundary-sensitive detector (vs vanilla detector) (ablation) | DiffDVC · YouCook2 | F1: +0.56 | Helped |
| increase number of sampling points K from 1 to 4 (ablation) | DiffDVC boundary-sensitive detector · ActivityNet Captions (TSN) | CIDEr: K=4 gives 29.90 vs 29.09 at K=1; K=8 gives 28.89 | Helped |
| increase number of proposals N from 15 to 100 (ablation) | DiffDVC · ActivityNet Captions (C3D/TSN/I3D) | CIDEr: +1.56 (C3D), +2.38 (TSN), +0.93 (I3D) | Mixed |
| lower signal scaling factor from 3.0 to 0.5 (ablation) | DiffDVC boundary-sensitive detector · ActivityNet Captions (I3D) | precision: precision drop of 1.13 for b-sensitive vs 2.07 for vanilla when factor goes 0.5 to 3.0 | Helped |
| use DiffDVC (diffusion with boundary-sensitive detector) instead of PDVC | dense video captioning · ActivityNet Captions (I3D) | SODA_c: 6.91 vs 5.42 | Helped |
| use DiffDVC instead of DiffTAD-style diffusion detection | event detection · ActivityNet-1.3 | mAP avg: 35.8 vs 34.8 | Helped |
| use DiffDVC instead of discriminative TriDet on ActivityNet-1.3 | event detection · ActivityNet-1.3 | mAP avg: 35.8 vs 36.8 | Hurt |
Surprising findings
- Event detection gains from the boundary-sensitive detector are modest, and DiffDVC does not beat the best discriminative detectors (e.g., TriDet) on ActivityNet-1.3.
- Shifting ground-truth proposals by only 5% sharply reduces captioning CIDEr (from 52.38 to 40.52 in the reported table), suggesting caption quality is highly sensitive to boundary accuracy.
Limitations
- Does not outperform the best discriminative event detectors on ActivityNet-1.3 average mAP.
- Evaluation uses validation sets because test sets are inaccessible.
- Gains on some metrics (e.g., BLEU@4 on C3D) are not consistent across feature types.
Experimental setup
Datasets: ActivityNet-1.3 ActivityNet Captions YouCook2 THUMOS14 COCO
Benchmarks: ActivityNet-1.3 ActivityNet Captions YouCook2
Metrics: mAP precision recall F1 BLEU@4 METEOR CIDEr SODA_c
Baselines: DiffTAD PDVC TadTR Actionformer TriDet Vid2Seq SDVC MFAM
Builds on: DiffusionDet DiffTAD PDVC Deformable DETR DDIM
Closest papers by meaning
- Event-Equalized Dense Video Captioning (CVPR 2025 · similarity 0.88)
- DIBS: Enhancing Dense Video Captioning with Unlabeled Videos via Pseudo Boundary Enrichment and Online Refinement (CVPR 2024 · similarity 0.88)
- Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.88)
- Video Anomaly Detection with Motion and Appearance Guided Patch Diffusion Model (AAAI 2025 · similarity 0.87)
- Action Detection via an Image Diffusion Process (CVPR 2024 · similarity 0.86)
- VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.