Research map / Video / Video-language understanding
Video ReCap: Recursive Captioning of Hour-Long Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Video ReCap, a recursive video-language model that generates hierarchical captions (clip, segment, hour-long summary) for videos from seconds to two hours, trained with curriculum learning and LLM-generated pseudo-annotations, and releases the Ego4D-HCap dataset.
Problem
Existing video captioning models handle only few-second clips and cannot produce multi-level captions for minutes-to-hours videos with hierarchical structure.
Approach
A recursive video-language architecture where each hierarchy level consumes sparsely sampled video features plus captions from the previous level, trained with a clip-to-segment-to-summary curriculum and augmented with FLAN-T5-generated pseudo summaries.
Key results
- Video ReCap outperforms LaViLa by 9.79% CIDEr on clip captioning, 17.11% on segment descriptions and 21.52% on video summaries on Ego4D-HCap.
- On EgoSchema long-range VideoQA, Video ReCap + GPT3.5 with hierarchical captions reaches 50.23% accuracy, 18.13% above the prior best InternVideo (32.1%).
- Curriculum learning improves video summary CIDEr by 19.44% over training from GPT2 initialization; LLM pseudo-annotations add 5.14% CIDEr on segment descriptions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use recursive video and text inputs rather than video-only input (ablation) | Video ReCap · Ego4D-HCap segment descriptions | CIDEr: 40.17 -> 41.74 | Helped |
| Use recursive video and text inputs rather than text-only input (ablation) | Video ReCap · Ego4D-HCap segment descriptions | CIDEr: 40.10 -> 41.74 | Helped |
| Use recursive video and text inputs rather than video-only input (ablation) | Video ReCap · Ego4D-HCap video summaries | CIDEr: 25.64 -> 28.06 | Helped |
| Use recursive video and text inputs rather than text-only input (ablation) | Video ReCap · Ego4D-HCap video summaries | CIDEr: 23.23 -> 28.06 | Helped |
| Train clip captions before segment descriptions (ablation) | Video ReCap · Ego4D-HCap segment descriptions | CIDEr: 36.81 -> 41.74 | Helped |
| Train clip captions before video summaries (ablation) | Video ReCap · Ego4D-HCap video summaries | CIDEr: 8.62 -> 24.84 | Helped |
| Progressively train clip captions, segment descriptions, then video summaries (ablation) | Video ReCap · Ego4D-HCap video summaries | CIDEr: 24.84 -> 28.06 | Helped |
| Add LLM-generated pseudo annotations (ablation) | Video ReCap · Ego4D-HCap segment descriptions | CIDEr: 41.74 -> 46.88 | Helped |
| Add LLM-generated pseudo annotations (ablation) | Video ReCap · Ego4D-HCap video summaries | CIDEr: 28.06 -> 29.34 | Helped |
| Use hierarchical captions instead of short-term captions for GPT3.5 VideoQA | Video ReCap + GPT3.5 · EgoSchema | QA accuracy: 46.03 -> 50.23 | Helped |
| Use Video ReCap hierarchical captions instead of LaViLa captions for GPT3.5 VideoQA | GPT3.5 · EgoSchema | QA accuracy: 44.27 -> 50.23 | Helped |
| Use Video ReCap rather than InternVideo | EgoSchema VideoQA · EgoSchema | QA accuracy: +18.13% | Helped |
Surprising findings
- Video ReCap-U with only 113M trainable parameters still outperforms the larger LaViLa baseline by about 21-25% CIDEr on segment and summary generation, suggesting gains come from the recursive design rather than capacity.
Limitations
- Hierarchical captioning is evaluated mainly on Ego4D egocentric videos, limiting generality to other domains.
- Long videos require sparse sampling of video features to fit GPU memory, discarding information.
- Summary and segment supervision relies on limited manual annotations (8,267 summaries) plus LLM pseudo-labels.
Experimental setup
Datasets: Ego4D Ego4D-HCap EgoSchema
Benchmarks: Ego4D-HCap EgoSchema
Metrics: CIDEr ROUGE-L METEOR accuracy
Baselines: BLIP2 BLIP2 + GPT3.5 LaViLa + GPT3.5 LaViLa + GPT2 LaViLa + FLAN-T5 LaViLa InternVideo EgoVLP EgoVLPv2
Builds on: LaViLa BLIP-2 Flamingo-style gated cross-attention Ego4D
Closest papers by meaning
- Fine-Grained Captioning of Long Videos through Scene Graph Consolidation (ICML 2025 · similarity 0.88)
- Vript: A Video Is Worth Thousands of Words (NeurIPS 2024 · similarity 0.88)
- Progress-Aware Video Frame Captioning (CVPR 2025 · similarity 0.88)
- AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark (ICLR 2025 · similarity 0.87)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.87)
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.