Research map / Video / Video-language understanding
CALVIN: Improved Contextual Video Captioning via Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CALVIN, a Llama-2-7B-based video LLM that uses preceding movie scene context and two-stage instruction tuning to generate contextual audio-description captions, with prompting and few-shot test-time adaptation.
Problem
Generating contextual scene captions (audio descriptions) for movies: existing VLMs produce verbose, superficial, and context-agnostic descriptions and hallucinate character names.
Approach
A video LLM with a frozen CLIP ViT-H/14 encoder and a Q-Former plus linear projection into Llama-2 7B, trained in two stages (image-caption alignment, then instruction tuning mixing image VQA, WebVid, and MAD with prior-scene text context), plus entity prompting and few-shot test-time fine-tuning.
Key results
- On MAD-named-eval with 5-scene context, CALVIN reaches CIDEr 27.71 and BertScore 40.18, versus AutoAD-II's CIDEr 19.5, a ~26% CIDEr gain over the prior SOTA.
- Zero-shot on TVC, CALVIN attains CIDEr 38.9 versus 12.4 for VideoLLaVA, roughly 3x the next-best model.
- Few-shot fine-tuning on 50 test-movie scenes raises CIDEr from 10.85 to 20.06 without context; entity prompting raises it to 16.27.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CALVIN contextual captioning with 3-scene context instead of the strongest reported contextual baseline | MAD-named-eval · MAD-named-eval, context | CIDEr: 21.9 -> 25.47 | Helped |
| Add CALVIN contextual captioning with 5-scene context instead of the 3-scene context | CALVIN · MAD-named-eval | CIDEr: 25.47 -> 27.71 | Helped |
| Use CALVIN instead of the strongest reported zero-shot baseline | TVC · TVC, zero-shot | CIDEr: 8.43 -> 38.9 | Helped |
| Add MAD data to Stage-2 training (ablation) | CALVIN · MAD-named-eval, context (3 scenes) | CIDEr: 3.52 -> 25.47 | Helped |
| Add WebVid data to Stage-2 training (ablation) | CALVIN · MAD-named-eval, context (3 scenes) | CIDEr: 22.07 -> 25.47 | Helped |
| Add image VQA data to Stage-2 training (ablation) | CALVIN · MAD-named-eval, context (3 scenes) | CIDEr: 20.21 -> 25.47 | Helped |
| Add Stage-2 training data (ablation) | CALVIN · MAD-named-eval, context (3 scenes) | CIDEr: 2.24 -> 25.47 | Helped |
| Train both Q-Former and LLM rather than only one component (ablation) | CALVIN · MAD-named-eval, context (3 scenes) | CIDEr: 15.91 -> 25.47; 0.1 -> 25.47 | Helped |
| Use 32 Q-Former tokens rather than 64 or 128 (ablation) | CALVIN · MAD-named-eval, context (3 scenes) | CIDEr: 24.27 -> 25.47; 23.10 -> 25.47 | Helped |
| Use LoRA instead of prefix tuning (ablation) | CALVIN · MAD-named-eval, context (3 scenes) | CIDEr: 11.46 -> 25.47 | Helped |
| Increase LoRA rank from 16 to 32 (ablation) | CALVIN · MAD-named-eval, context (3 scenes) | CIDEr: 24.67 -> 25.47 | Helped |
| Increase LoRA rank from 32 to 64 (ablation) | CALVIN · MAD-named-eval, context (3 scenes) | CIDEr: 25.47 -> 24.73 | No clear effect |
Surprising findings
- Training only the LLM (Q-Former frozen) collapses performance (CIDEr 0.1), suggesting the model ignores vision and attends only to text context.
- Mixing MAD into stage 2 matched or beat a separate stage-3 MAD fine-tune, contrary to the AutoAD recipe.
Limitations
- Single training run per configuration, so no error bars or significance tests.
- Few-shot fine-tuning can reduce caption diversity and length and sometimes associates wrong names with characters.
- MAD provides only CLS embeddings and no raw video, limiting comparisons with general video LLMs to zero-shot TVC evaluation.
Experimental setup
Datasets: MAD CC-12M CC-3M LLaVA-1.5 instruction data WebVid-3M OKVQA A-OKVQA TextCaps Visual Genome
Benchmarks: MAD-named-eval TV-Captioning (TVC)
Metrics: BertScore CIDEr ROUGE-L SPICE
Baselines: ClipCap CapDec AutoAD-I AutoAD-II VideoLlama MovieChat VideoLLaVA
Builds on: LLaVA Q-Former AutoAD-II LoRA
Closest papers by meaning
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.89)
- ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentation (CVPR 2025 · similarity 0.87)
- Fine-Grained Captioning of Long Videos through Scene Graph Consolidation (ICML 2025 · similarity 0.87)
- Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning (CVPR 2025 · similarity 0.87)
- Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.87)
- FlexCap: Describe Anything in Images in Controllable Detail (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.