Research map / Video / Video-language understanding
Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Chapter-Llama fine-tunes Llama-3.1-8B-Instruct with LoRA on ASR transcripts and speech-guided frame captions to generate timestamped chapters for hour-long videos, improving VidChapters-7M F1 from 26.7 to 45.3.
Problem
Automatic video chaptering of hour-long videos: partitioning a long video into semantic segments with timestamps and titles, which existing end-to-end video models handle poorly given limited frames and visual-to-text mapping needs.
Approach
Cast chaptering as a text-only task: convert speech transcripts and captions of speech-selected keyframes into timestamped text, and fine-tune a long-context LLM to output chapter boundaries and titles in one pass, with iterative windowing for very long inputs.
Key results
- Chapter-Llama reaches 45.3 F1 and 19.3 SODA on the full VidChapters-7M test set versus 26.7 F1 and 11.6 SODA for Vid2Seq.
- Speech-based frame selection captions about 10.3 frames per video and outperforms 100 equidistant frames on segmentation (42.6 vs 39.2 F1 with speech+captions).
- Zero-shot Chapter-Llama reaches 29.5 F1 without any chapter annotations, while Vid2Seq trained only on HowTo100M gets 3.0 F1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Finetune the LLM on chapter annotations (ablation) | Chapter-Llama with speech and captions · VidChapters-7M validation subset | F1: 29.9 -> 42.6 | Helped |
| Add captions to speech input (ablation) | Finetuned Chapter-Llama · VidChapters-7M validation subset | F1: 38.5 -> 42.6 | Helped |
| Add speech input to captions (ablation) | Finetuned Chapter-Llama · VidChapters-7M validation subset | F1: 39.1 -> 42.6 | Helped |
| Use speech-based frame selection instead of alternative sampling strategies (ablation) | Finetuned Chapter-Llama with speech and captions · VidChapters-7M validation subset | F1: 39.2 -> 42.6; 41.0 -> 42.6; 40.6 -> 42.6; 40.1 -> 42.6 | Helped |
| Replace captions with SigLIP frame embeddings (ablation) | Chapter-Llama with speech input · VidChapters-7M validation subset | F1: 42.6 -> 40.4 | Hurt |
| Add SigLIP frame embeddings to speech and captions (ablation) | Chapter-Llama · VidChapters-7M validation subset | F1: 42.6 -> 44.4 | Helped |
| Process long videos iteratively instead of using only the first input window (ablation) | Chapter-Llama, videos exceeding 35k tokens · VidChapters-7M validation subset | F1: 13.1 -> 18.5; 16.6 -> 23.6; 18.7 -> 25.3 | Helped |
| Use Chapter-Llama instead of Vid2Seq | VidChapters-7M test set · VidChapters-7M test (All) | F1: 26.7 -> 45.3 | Helped |
| Use Chapter-Llama instead of Gemini-1.5-Pro in zero-shot evaluation | VidChapters-7M test set · VidChapters-7M test (All) | F1: 42.2 -> 45.3 | Helped |
Surprising findings
- Zero-shot prompting with long multimodal text degrades performance, so fine-tuning is needed to exploit the context.
- Captions alone are weaker than speech alone for chaptering, and the speech-only LLM is used to select frames for captioning.
Limitations
- Performance depends on the accuracy of the ASR and visual captioner.
- Context window limits (about 15k training tokens, 25k inference tokens) require iterative chunked prediction for very long videos.
- Trained on a 20k-video subset, and biases from web-trained models may cause inaccurate chaptering for underrepresented topics.
Experimental setup
Datasets: VidChapters-7M
Benchmarks: VidChapters-7M
Metrics: F1 tIoU SODA (S) CIDEr (C)
Baselines: Vid2Seq GPT-4o GPT-4o-mini Gemini-1.5-Pro Gemini-2.0-Flash shot detection
Builds on: Vid2Seq LoRA Llama 3.1 WhisperX SCSampler
Closest papers by meaning
- World Model on Million-Length Video And Language With Blockwise RingAttention (ICLR 2025 · similarity 0.87)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.87)
- Video Summarization with Large Language Models (CVPR 2025 · similarity 0.87)
- VISTA-LLAMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokens (CVPR 2024 · similarity 0.87)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.87)
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.