Research map / Video / Video-language understanding
DIBS: Enhancing Dense Video Captioning with Unlabeled Videos via Pseudo Boundary Enrichment and Online Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pretrains dense video captioning models on ~56k unlabeled HowTo100M cooking videos using LLM-generated event captions, optimized pseudo boundaries, and online boundary refinement built on PDVC.
Problem
Dense video captioning needs precise event boundaries and captions, which are scarce and expensive to annotate; prior large-scale pretraining (Vid2Seq) uses noisy raw subtitles as pseudo captions and boundaries.
Approach
Use diverse LLMs to rewrite subtitles into event-centric caption candidates, generate pseudo boundaries via caption-aware frame-similarity optimization with soft time constraints, and iteratively refine boundaries online during training using PDVC proposal scores.
Key results
- With UniVL backbone, DIBS reaches 9.41 METEOR and 59.35 CIDEr on YouCook2, surpassing Vid2Seq (9.30 / 47.10 with CLIP) on captioning metrics.
- DIBS uses about 56k unlabeled videos versus Vid2Seq's 15M (about 0.4%).
- On ActivityNet, the CLIP-based DIBS achieves 8.93 METEOR and 31.89 CIDEr, but localization lags behind PDVC and UEDVC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DIBS pretraining instead of Vid2Seq | DVC with CLIP backbone · YouCook2 | CIDEr: 47.10 -> 44.44 | Helped |
| Use DIBS pretraining instead of Vid2Seq | DVC with CLIP backbone · ActivityNet | METEOR: 8.50 -> 8.93 | Helped |
| Use DIBS pretraining instead of PDVC | DVC with UniVL backbone · YouCook2 | CIDEr: 46.02 -> 59.35 | Helped |
| Use DIBS pretraining instead of PDVC | DVC with CLIP backbone · ActivityNet | CIDEr: 30.11 -> 31.89 | Helped |
| Add generated pseudo boundaries (ablation) | DIBS with CLIP backbone · YouCook2 | CIDEr: 9.07 -> 21.93 | Helped |
| Add soft time constraints (ablation) | DIBS pseudo-boundary generation · YouCook2 | CIDEr: 25.57 -> 28.04 | Helped |
| Add soft time constraints (ablation) | DIBS pseudo-boundary generation · ActivityNet | CIDEr: 15.23 -> 16.63 | Helped |
| Add online pseudo-boundary refinement (ablation) | DIBS with HowTo100M pretraining · YouCook2 | CIDEr: 54.98 -> 59.35 | Helped |
| Add online pseudo-boundary refinement (ablation) | DIBS without HowTo100M pretraining · ActivityNet | SODA c: 4.68 -> 4.67 | No clear effect |
| Add HowTo100M pretraining (ablation) | DIBS with UniVL backbone · YouCook2 | CIDEr: 46.02 -> 59.35 | Helped |
| Add HowTo100M pretraining (ablation) | DIBS with UniVL backbone · ActivityNet | CIDEr: 28.21 -> 28.85 | No clear effect |
| Increase fine-tuning data from 25% to 100% (ablation) | DIBS after pretraining · YouCook2 | CIDEr: 46.69 -> 59.35 | Helped |
Surprising findings
- Pretraining improves captioning more consistently than event localization; localization gains are not stable, especially on ActivityNet.
- Backbone preference depends on dataset: UniVL is better on YouCook2 while CLIP is better on ActivityNet.
Limitations
- Substantial domain gap between instructional videos and activity videos hurts ActivityNet localization.
- Pseudo boundaries still contain noise and remain far from ground-truth boundaries, limiting localization gains from pretraining.
- Prompt design involved manual correction of LLM errors in the loop.
Experimental setup
Datasets: HowTo100M YouCook2 ActivityNet Captions
Benchmarks: YouCook2 ActivityNet Captions
Metrics: METEOR CIDEr SODA_c Precision Recall F1
Baselines: Vid2Seq PDVC UEDVC E2ESG WLT WS-DEC EC-SL
Builds on: PDVC Vid2Seq Drop-DTW UniVL CLIP
Closest papers by meaning
- Event-Equalized Dense Video Captioning (CVPR 2025 · similarity 0.88)
- DiffDVC: Accurate Event Detection for Dense Video Captioning via Diffusion Models (AAAI 2025 · similarity 0.88)
- Fine-Grained Captioning of Long Videos through Scene Graph Consolidation (ICML 2025 · similarity 0.87)
- VidChain: Chain-of-Tasks with Metric-based Direct Preference Optimization for Dense Video Captioning (AAAI 2025 · similarity 0.87)
- HiCM²: Hierarchical Compact Memory Modeling for Dense Video Captioning (AAAI 2025 · similarity 0.87)
- Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.