Research map / Video / Video-language understanding
Learnability Matters: Active Learning for Video Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a learnability-, diversity-, and uncertainty-aware active learning method with a caption-wise annotation protocol for video captioning, using BLIP2 per-frame captions as proxy ground truths.
Problem
Active learning for video captioning suffers from collective outliers (hard-to-learn samples caused by inconsistent human captions), which existing AL methods ignore because ground truths are unavailable during selection.
Approach
Estimate learnability by measuring consistency of approximate ground truths (BLIP2 per-frame captions and the video model's own captions, compared via SPICE scene graphs), combine it with TF-IDF-style diversity and uncertainty scores, and allocate human annotation at the caption level rather than the video level.
Key results
- On MSR-VTT with SwinBERT, the method reaches about 103% of full-data CIDEr performance with 25% of human annotations.
- On MSVD with SwinBERT, the method surpasses full-data performance on 4 of 5 metrics with under 25% annotations; CIDEr reaches 103.65% at 25%.
- Ablation on MSVD: AUC CIDEr rises from 0.574 (random) to 0.738 with the full method including caption-wise protocol.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed active learning selection instead of random selection with 20% of MSR-VTT annotations | SwinBERT trained on MSVD with MSR-VTT data · MSVD test | CIDEr: 123.26 -> 127.41 | Helped |
| Select MSR-VTT annotations iteratively rather than directly selecting 20% | Ours, SwinBERT trained on MSVD with MSR-VTT data · MSVD test | CIDEr: 127.41 -> 130.96 | Helped |
| Use uncertainty-based selection rather than random sampling (ablation) | Active learning on MSVD · MSVD | AUC CIDEr: .574 -> .583 | Helped |
| Add diversity to uncertainty-based selection (ablation) | Active learning on MSVD · MSVD | AUC CIDEr: .583 -> .589 | Helped |
| Add learnability term L2 (ablation) | Active learning on MSVD · MSVD | AUC CIDEr: .589 -> .594 | Helped |
| Add learnability term L1 (ablation) | Active learning on MSVD · MSVD | AUC CIDEr: .594 -> .603 | Helped |
| Add the caption-wise protocol (ablation) | Active learning on MSVD · MSVD | AUC CIDEr: .603 -> .738 | Helped |
| Add the caption-wise protocol (ablation) | Active learning on MSVD · MSVD | AUC SPICE: .600 -> .678 | Helped |
| Use BLIP2 captions as pseudo ground truths for semi-supervised training | SwinBERT · MSR-VTT test | CIDEr: 0.31 | Hurt |
Surprising findings
- Directly using BLIP2 per-frame captions as pseudo ground truths for semi-supervised training degrades performance below the starting point (CIDEr 0.31, SPICE 0.07).
- Training SwinBERT on MSR-VTT with more captions per video (up to full 20) shows performance degradation at full annotation, which the authors attribute to collective outliers rather than overfitting.
Limitations
- Authors say the paper only briefly connects curriculum learning and learnability.
- CIDEr does not always align with human evaluation; more human analysis is needed.
- Semi-supervised use of LVLM pseudo labels gave no significant improvement; ChatGPT-based consistency analysis is preliminary.
Experimental setup
Datasets: MSVD MSR-VTT
Benchmarks: MSVD MSR-VTT
Metrics: BLEU4 METEOR ROUGE-L CIDEr SPICE
Baselines: Random Sampling Maximum Entropy Minimum Likelihood Core-Set Selection Clustered-Divergence
Builds on: Dataset Maps SPICE Coreset curriculum learning SwinBERT
Closest papers by meaning
- Hierarchical Multi-Source Uncertainty Aggregation for Interactive Video Captioning (AAAI 2025 · similarity 0.90)
- Vript: A Video Is Worth Thousands of Words (NeurIPS 2024 · similarity 0.85)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.85)
- CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.85)
- CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning (ICLR 2026 · similarity 0.84)
- Cycle-Consistency Learning for Captioning and Grounding (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.