Research map / Video / Video-language understanding
NarrativeBridge: Enhancing Video Captioning with Causal-Temporal Narrative
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the CTN captions benchmark (LLM-generated cause-effect captions for MSR-VTT and MSVD) and the Cause-Effect Network (CEN), a two-stage model with separate cause and effect CLIP-based video encoders that outperforms video captioning SOTA and fine-tuned VLMs.
Problem
Existing video captioning benchmarks and models lack causal-temporal narrative, i.e., event sequences linked by cause and effect, so generated captions fail to capture causal and temporal dynamics.
Approach
Augment MSR-VTT and MSVD with LLM-generated cause-effect captions (Mixtral, few-shot prompting, EMScore filtering), then train a two-stage network where separate CLIP-ViT cause and effect video encoders are pretrained contrastively on cause and effect text and their frozen features feed a Uni-VL-initialized caption decoder.
Key results
- CEN achieves 17.88 CIDEr gain over GIT on MSVD-CTN (63.51 vs 45.63) and 17.44 gain on MSRVTT-CTN (49.87 vs 32.43).
- CEN reaches 27.90 ROUGE-L and 15.76 SPICE on MSRVTT-CTN, beating fine-tuned VideoLLaVA (22.18 R-L, 19.98 CIDEr) and ShareGPT4Video.
- Human evaluation of CTN captions gives a mean quality score of 4.8/5 (sigma 0.40) with ICC 0.87.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CEN instead of GIT | Video captioning on MSVD-CTN · MSVD-CTN | ROUGE-L: 3.95 ROUGE-L | Helped |
| Use CEN instead of GIT | Video captioning on MSVD-CTN · MSVD-CTN | CIDEr: 17.88 CIDEr | Helped |
| Use CEN instead of GIT | Video captioning on MSVD-CTN · MSVD-CTN | SPICE: 3.67 SPICE points | Helped |
| Use CEN instead of GIT | Video captioning on MSRVTT-CTN · MSRVTT-CTN | ROUGE-L: 3.39 ROUGE-L | Helped |
| Use CEN instead of GIT | Video captioning on MSRVTT-CTN · MSRVTT-CTN | CIDEr: 17.44 CIDEr | Helped |
| Use CEN instead of GIT | Video captioning on MSRVTT-CTN · MSRVTT-CTN | SPICE: 2.06 SPICE points | Helped |
| Use separate cause and effect video encoders instead of one combined encoder (ablation) | CEN · MSVD-CTN and MSRVTT-CTN | captioning metrics | Helped |
| Fine-tune the single CLIP encoder on CTN captions (ablation) | CEN · MSVD-CTN and MSRVTT-CTN | captioning metrics | Helped |
| Fine-tune two CLIP encoders on cause and effect captions (ablation) | CEN · MSVD-CTN and MSRVTT-CTN | captioning metrics | Helped |
| Add the effect encoder alongside the cause encoder (ablation) | CEN · MSVD-CTN and MSRVTT-CTN | captioning metrics | Helped |
| Add the cause encoder alongside the effect encoder (ablation) | CEN · MSVD-CTN and MSRVTT-CTN | captioning metrics | Helped |
| Use few-shot examples in the caption-generation prompt instead of zero-shot prompting (ablation) | CTN caption generation · Video caption generation | caption quality | Helped |
Surprising findings
- Zero-shot VLMs score far below the simple captioning baselines on CTN captions, and fine-tuning only partially closes the gap.
- Zero-shot prompt variants with missing requirements produced verbose, speculative, or temporally inconsistent captions, showing few-shot examples matter for generation quality.
Limitations
- Complex or convoluted causal relationships remain challenging.
- Architecture may need enhancement for generalization; spatial reasoning, multi-agent interactions and long-term dependencies are not explicitly modeled.
- Only 52.1% of CTN captions pass the EMScore threshold after iterative refinement, and the CTN captions are LLM-generated from existing captions rather than from video content.
Experimental setup
Datasets: MSR-VTT MSVD
Benchmarks: MSVD-CTN MSRVTT-CTN
Metrics: CIDEr ROUGE-L SPICE EMScore
Baselines: SEM-POS AKGNN GIT VideoLLaVA ShareGPT4Video
Builds on: CLIP4Clip CLIP Uni-VL EMScore
Closest papers by meaning
- Progress-Aware Video Frame Captioning (CVPR 2025 · similarity 0.87)
- EventLens: Event-Structure Reinforcement Learning for Video Understanding (NeurIPS 2026 · similarity 0.87)
- Two Causally Related Needles in a Video Haystack (NeurIPS 2025 · similarity 0.87)
- VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models (CVPR 2025 · similarity 0.86)
- Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Grounding (CVPR 2025 · similarity 0.86)
- Video ReCap: Recursive Captioning of Hour-Long Videos (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.