Research map / Video / Video-language understanding
Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SGVC, a training-free long-video captioning framework that turns segment captions from off-the-shelf VLMs into scene graphs, merges them by Hungarian-matched consolidation, and decodes the global graph with a lightweight graph-to-text model.
Problem
Zero-shot captioning of long videos beyond the temporal receptive field of short-clip VLMs; existing consolidation methods need supervised fine-tuning or costly LLM-based aggregation.
Approach
Consolidate segment-level scene graphs (parsed from per-segment captions) by iteratively merging the most similar graph pairs via Hungarian matching of graph-encoder object embeddings, then generate the video caption with a small graph-to-text decoder trained only on text/graph-text pairs.
Key results
- On MSR-VTT zero-shot captioning, SGVC with BLIP2 reaches CIDEr 26.1 and B@4 18.4, versus 16.5 CIDEr for Video ChatCaptioner and 15.4 CIDEr for Mistral-7B summarization with BLIP2 captions.
- On MSVD, SGVC with BLIP2 reaches B@4 25.3 and CIDEr 53.3 and F_BERT 0.597, above Video ChatCaptioner (CIDEr 35.8).
- On ActivityNet ae-val paragraph captioning, SGVC with InternVL2.5 reaches CIDEr 24.1 and F_BERT 0.338, above GPT-4o mini summarization (CIDEr 15.3) and Video ChatCaptioner (CIDEr 1.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Mistral-7B summarization of segment captions with scene-graph consolidation | SGVC with BLIP2 · MSR-VTT test | CIDEr: 15.4 -> 26.1 | Helped |
| replace Mistral-7B summarization of segment captions with scene-graph consolidation | SGVC with BLIP2 · MSVD test | CIDEr: 41.6 -> 53.3 | Helped |
| replace GPT-4o mini summarization of segment captions with scene-graph consolidation | SGVC with InternVL2.5 · ActivityNet Captions ae-val | CIDEr: 15.3 -> 24.1 | Helped |
| use prioritized subgraph extraction with k=1 instead of k=3 (ablation) | SGVC · MSR-VTT test | CIDEr: 24.9 -> 26.1 | Helped |
| use prioritized subgraph extraction with k=3 instead of k=1 (ablation) | SGVC · MSR-VTT test | METEOR: 23.1 -> 23.8 | Helped |
| use prioritized subgraph extraction with k=1 instead of k=3 (ablation) | SGVC · MSR-VTT test | P_BERT: 0.454 -> 0.467 | Helped |
| use prioritized subgraph extraction with k=3 instead of k=1 (ablation) | SGVC · MSR-VTT test | R_BERT: 0.542 -> 0.554 | Helped |
| use prioritized subgraph extraction with k=1 instead of k=3 (ablation) | SGVC · MSR-VTT test | F_BERT: 0.486 -> 0.487 | Helped |
| vary graph consolidation threshold from τ=0.80 to τ=0.95 (ablation) | SGVC · MSVD test | CIDEr: 49.9; 50.0; 49.9; 50.2 | No clear effect |
| vary graph consolidation threshold from τ=0.80 to τ=0.95 (ablation) | SGVC · MSVD test | F_BERT: 0.589; 0.589; 0.589; 0.589 | No clear effect |
Surprising findings
- VidIL with few-shot exemplars from reference captions is competitive with or higher than SGVC on some metrics, though SGVC is zero-shot.
- The graph-to-text decoder is trained only on image caption and text corpora, yet it transfers to video captioning without video-specific training.
Limitations
- Errors from hallucinated segment-level captions propagate to the final caption (e.g., misidentified objects, wrong counts).
- Comparisons against supervised or fine-tuned long-video methods are not made; evaluation focuses on LLM-based zero-shot approaches.
- Scene graph merging runs on CPU and is not optimized; main metrics are n-gram based despite BERTScore addition.
Experimental setup
Datasets: MS-COCO Flickr30k TextCaps Visual Genome Visual Genome Paragraph Captions Kinetics-400 MSR-VTT MSVD ActivityNet Captions
Benchmarks: MSR-VTT MSVD ActivityNet Captions ae-val
Metrics: BLEU-4 METEOR CIDEr BERTScore
Baselines: VidIL Video ChatCaptioner LLM summarization with Mistral-7B-Instruct-v0.3 LLM summarization with GPT-4o mini ZeroCap MAGIC DeCap C3 Tewel et al.
Builds on: FACTUAL-MR BERT T5 VidIL Video ChatCaptioner
Closest papers by meaning
- Progress-Aware Video Frame Captioning (CVPR 2025 · similarity 0.90)
- Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning (CVPR 2025 · similarity 0.89)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.89)
- Video ReCap: Recursive Captioning of Hour-Long Videos (CVPR 2024 · similarity 0.88)
- Weakly Supervised Video Scene Graph Generation via Natural Language Supervision (ICLR 2025 · similarity 0.88)
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.