Research map / Video / Video-language understanding
LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a Video LLM (LiveCC-7B) by densely interleaving timestamped ASR words with video frames on 10M YouTube clips, then SFT on 408K WhisperX-transcribed clips, and introduces Sports-3K for real-time commentary evaluation.
Problem
Video LLMs rely on costly human annotations or proprietary API-generated data, limiting scale; streaming video understanding and real-time commentary lack scalable training data and evaluation.
Approach
Treat cheap ASR transcripts as time-aligned word streams and interleave ASR words with per-frame visual tokens in an autoregressive streaming sequence, so the model learns temporally aligned vision-language modeling and can generate short commentary per frame with low latency.
Key results
- LiveCC-7B reaches 63.9 overall on Video-MME, outperforming Qwen2-VL-7B-Instruct (63.3) and LLaVA-Video-7B (63.3) at the 7B scale.
- On Sports-3K-CC, streaming pre-training on 5M clips reaches a 36.90% win rate against LLaVA-Video-72B, and 10M clips reach 41.44%.
- LiveCC-7B scores 66.10 on Sports-3K-QA overall, comparable to LLaVA-Video-7B (66.44) and above LLaVA-Video-72B on OCR questions (65.77 vs 65.10).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use streaming rather than caption-style pre-training (ablation) | LiveCC · Sports-3K-CC | win rate vs LLaVA-Video-72B: 14.42 -> 36.90 | Helped |
| Use streaming rather than caption-style pre-training (ablation) | LiveCC · Video-MME | accuracy: 61.1 -> 61.0 | No clear effect |
| Add previous ASR text as context (ablation) | LiveCC · Sports-3K-CC | win rate vs LLaVA-Video-72B: 16.45 -> 36.14 | Helped |
| Add previous ASR text as context (ablation) | LiveCC · Video-MME | accuracy: 60.7 -> 61.1 | Helped |
| Add video title as context (ablation) | LiveCC · Sports-3K-CC | win rate vs LLaVA-Video-72B: 16.45 -> 25.66 | Helped |
| Add video title as context (ablation) | LiveCC · Video-MME | accuracy: 60.7 -> 59.7 | Hurt |
| Use title only when previous ASR is unavailable (ablation) | LiveCC · Video-MME | accuracy: 60.7 -> 61.0 | Helped |
| Increase streaming pre-training data from 5M to 10M clips (ablation) | LiveCC · Sports-3K-CC | win rate vs LLaVA-Video-72B: 36.90 -> 41.44 | Helped |
| Increase streaming pre-training data from 5M to 10M clips (ablation) | LiveCC · Video-MME | accuracy: 61.0 -> 58.0 | Hurt |
Surprising findings
- Pre-training on streaming ASR alone (no SFT) already yields strong general video QA (Video-MME above 60), exceeding many SFT models.
- Simply adding video titles as context degraded Video-MME performance, attributed to possible information leakage.
Limitations
- YouTube closed captions are low quality, so pre-training relies on noisy transcripts; SFT uses costly WhisperX re-transcription.
- Pre-training on single-source streaming commentary data degrades QA beyond 5M clips, motivating multi-source SFT.
- Sports-3K-CC evaluation relies on GPT-4o-mini as pairwise judge against LLaVA-Video-72B.
Experimental setup
Datasets: HD-VILA YT-Temporal-1B VidChapters HowTo100M Live-CC-10M Live-WhisperX-408K LLaVA-Video-178K
Benchmarks: Video-MME Sports-3K-QA Sports-3K-CC
Metrics: accuracy win rate
Baselines: GPT-4o Gemini-1.5-Pro VideoLLaMA-2-72B Qwen2-VL-72B-Instruct LLaVA-OV-72B LLaVA-Video-72B LongVA-7B InternLM-XC2.5-7B Oryx-7B LLaVA-OV-7B Qwen2-VL-7B-Instruct LLaVA-Video-7B
Builds on: Qwen2-VL LLaVA-Video Vid2Seq WhisperX
Closest papers by meaning
- VideoLLM-online: Online Video Large Language Model for Streaming Video (CVPR 2024 · similarity 0.90)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.88)
- LiveStar: Live Streaming Assistant for Real-World Online Video Understanding (NeurIPS 2025 · similarity 0.87)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.87)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.87)
- Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.