Research map / Video / Video-language understanding
AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AuroraCap, a LLaVA-style video captioner that uses token merging to cut visual tokens by 80-90%, plus the VDC detailed-captioning benchmark and the LLM-based VDCscore metric.
Problem
Video detailed captioning lacks benchmarks with long, structured captions and evaluation metrics that reliably judge detailed captions; lengthy video token sequences also make LMM video captioning expensive.
Approach
Apply training-time frame-wise token merging (bipartite soft matching) to a LLaVA-style LMM with no extra temporal parameters, pair it with a three-stage training recipe, and evaluate detailed captions via a divide-and-conquer QA-based LLM metric (VDCscore) on a new benchmark of about 1,027 long structured captions.
Key results
- AuroraCap-7B reaches CIDEr 88.9 on Flickr30k zero-shot, versus 55.3 for GPT-4V and 82.2 for Gemini-1.5 Pro as reported in the paper.
- On the VDC benchmark, AuroraCap-7B attains the top VDCscore accuracy (38.21) and score (1.98) among the compared models, with Elo 1,267 versus 1,171 for Gemini-1.5 Pro.
- Token merging keeps 10% of visual tokens in the language stage with marginal performance drop, and the paper reports over 80% performance retained at about 0.2–0.4 kept ratio.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AuroraCap-7B instead of Gemini-1.5 Pro | Image captioning · Flickr | CIDEr: 82.2 -> 88.9 | Helped |
| Use AuroraCap-7B instead of InstructBLIP-7B | Image captioning · NoCaps | CIDEr: 118.2 -> 111.4 | Hurt |
| Use AuroraCap-7B instead of InstructBLIP-7B | Image captioning · COCO-Cap | CIDEr: 141.3* -> 120.8 | Hurt |
| Use AuroraCap-7B instead of the strongest listed baseline | Video captioning · MSR-VTT | CIDEr: 22.1 -> 33.1 | Helped |
| Use AuroraCap-7B instead of the strongest listed baseline | Video captioning · VATEX | CIDEr: 23.9 -> 33.8 | Helped |
| Use AuroraCap-7B instead of InternVL-2-8B | Video detailed captioning · VDC | VDCscore accuracy: 37.72 -> 38.21 | Helped |
| Use AuroraCap-7B instead of MovieChat+ | Video question answering · MSVD-QA | accuracy: 76.5 -> 62.6 | Hurt |
| Add full first-frame visual tokens (slowfast inference) (ablation) | AuroraCap-7B · MSR-VTT | CIDEr: 26.72 -> 26.18 | Hurt |
| Add full first-frame visual tokens (slowfast inference) (ablation) | AuroraCap-7B · ActivityNet-QA | accuracy: 58.55 -> 59.66 | Helped |
| Add full first-frame visual tokens (slowfast inference) (ablation) | AuroraCap-7B · MSVD-QA | accuracy: 56.45 -> 55.65 | Hurt |
| Add full first-frame visual tokens (slowfast inference) (ablation) | AuroraCap-7B · MSRVTT-QA | accuracy: 37.26 -> 38.22 | Helped |
| Reduce visual-token retention through token merging (ablation) | AuroraCap · Video captioning benchmarks | performance | Mixed |
Surprising findings
- Token merging to very low kept ratios sometimes improves video captioning scores (e.g., MSR-VTT, VATEX, ActivityNet-QA at ratios below 0.1).
- Mixing lower and higher visual token ratios during training gave no significant advantage, and slowfast inference gave marginal or negative gains.
Limitations
- Performs poorly on OCR-related tasks due to limited OCR training samples.
- Weak on camera handling and detailed captioning per the authors' own conclusion, and poor trade-off between performance and input token scale at current stage.
- Video QA on MSVD and MSR-VTT is modest relative to dedicated video models.
Experimental setup
Datasets: LAION-CC-SBU-595K ShareGPT4V DenseFusion MiraData LLaVA-Hound ShareGPT4Video LLaVA-Mix-665K M3IT Panda-70M Ego4D
Benchmarks: Flickr30k NoCaps COCO-Cap MSR-VTT VATEX MSVD-QA ActivityNet-QA MSRVTT-QA iVQA VDC
Metrics: CIDEr BLEU-4 METEOR ROUGE-L VDCscore accuracy VDCscore score Elo
Baselines: Gemini-1.5 Pro GPT-4V LLaVA-1.5 LLaVA-NeXT-Video InternVL-2-8B LLaVA-OV-7B ShareGPT4Video-8B MovieChat
Builds on: LLaVA Token Merging (ToMe) Prismatic VLMs Video-ChatGPT evaluation
Closest papers by meaning
- Progress-Aware Video Frame Captioning (CVPR 2025 · similarity 0.89)
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.89)
- Vript: A Video Is Worth Thousands of Words (NeurIPS 2024 · similarity 0.89)
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · similarity 0.89)
- InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption (CVPR 2025 · similarity 0.88)
- Fine-Grained Captioning of Long Videos through Scene Graph Consolidation (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.