Research map / Video / Video-language understanding
Hierarchical Multi-Source Uncertainty Aggregation for Interactive Video Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes hierarchical multi-source uncertainty aggregation (syntax, concept, alignment) built on conditional vacuity from evidential learning to select informative videos for active video captioning.
Problem
Entropy-based uncertainty for generative video captioning is inflated and ignores the diversity of valid captions, which misleads active sample selection for annotation in specialized domains.
Approach
Extends evidential-learning vacuity to a word-level conditional vacuity, then hierarchically aggregates it over words (syntax), sampled tags (concept), and CLIP-based video-caption alignment into a holistic uncertainty used to rank unlabeled videos for annotation.
Key results
- On MSR-VTT the proposed method reaches BLEU-4 0.252, ROUGE 0.493, CIDEr 0.431, versus best baseline MSSE at 0.239, 0.470, 0.417.
- On UCFCAP the proposed method reaches CIDEr 0.425 versus 0.406 for MSSE.
- Ablations show removing syntax, concept, or alignment uncertainty each lowers scores relative to the full aggregation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hierarchical multi-source uncertainty aggregation | Proposed video captioning framework · UCFCAP | ROUGE: 0.342 -> 0.359 | Helped |
| Use hierarchical multi-source uncertainty aggregation | Proposed video captioning framework · MSRVTT | ROUGE: 0.470 -> 0.493 | Helped |
| Add syntax-level uncertainty rather than using negative likelihood (ablation) | Proposed video captioning framework · UCFCAP | ROUGE: 0.354 -> 0.359 | Helped |
| Add syntax-level uncertainty rather than using negative likelihood (ablation) | Proposed video captioning framework · MSRVTT | ROUGE: 0.462 -> 0.493 | Helped |
| Add aggregation over tags (ablation) | Proposed video captioning framework · UCFCAP | ROUGE: 0.349 -> 0.359 | Helped |
| Add aggregation over tags (ablation) | Proposed video captioning framework · MSRVTT | ROUGE: 0.454 -> 0.493 | Helped |
| Add alignment between visual and text embeddings (ablation) | Proposed video captioning framework · UCFCAP | ROUGE: 0.340 -> 0.359 | Helped |
| Add alignment between visual and text embeddings (ablation) | Proposed video captioning framework · MSRVTT | ROUGE: 0.451 -> 0.493 | Helped |
| Use the proposed decoding strategy instead of greedy decoding (ablation) | Proposed video captioning framework · UCFCAP | ROUGE: 0.350 -> 0.359 | Helped |
| Use the proposed decoding strategy instead of greedy decoding (ablation) | Proposed video captioning framework · MSRVTT | ROUGE: 0.464 -> 0.493 | Helped |
| Use the proposed decoding strategy instead of beam search (ablation) | Proposed video captioning framework · UCFCAP | ROUGE: 0.352 -> 0.359 | Helped |
| Use the proposed decoding strategy instead of beam search (ablation) | Proposed video captioning framework · MSRVTT | ROUGE: 0.485 -> 0.493 | Helped |
Surprising findings
- Entropy-based uncertainty stays high even for captions with high ROUGE scores, indicating inflation for captioning.
Limitations
- Relies on Monte-Carlo sampling of captions and tags, adding computational overhead.
- Baselines for uncertainty-based video captioning are limited, so comparison is narrow.
- Evaluated only on two datasets with small improvements over baselines.
Experimental setup
Datasets: UCFCAP MSR-VTT
Benchmarks: UCFCAP MSR-VTT
Metrics: BLEU-4 ROUGE CIDEr
Baselines: JLL SE CRE MSSE
Builds on: evidential learning Perceiver resampler CLIP GPT-2
Closest papers by meaning
- Learnability Matters: Active Learning for Video Captioning (NeurIPS 2024 · similarity 0.90)
- Towards Understanding and Quantifying Uncertainty for Text-to-Image Generation (CVPR 2025 · similarity 0.85)
- Progress-Aware Video Frame Captioning (CVPR 2025 · similarity 0.84)
- CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation (NeurIPS 2026 · similarity 0.83)
- Vript: A Video Is Worth Thousands of Words (NeurIPS 2024 · similarity 0.83)
- A Bias-Variance-Covariance Decomposition of Kernel Scores for Generative Models (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.