Research map / Video / Video-language understanding
VideoLLM-online: Online Video Large Language Model for Streaming Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LIVE, a framework with a streaming EOS prediction objective, offline-to-streaming dialogue data generation, and a parallel KV-cached inference pipeline, instantiated as VideoLLM-online-7B for real-time video streaming dialogue.
Problem
Existing video LMMs process predetermined short clips offline and cannot handle temporally aligned, long-context, real-time dialogue over continuous streaming video; per-frame chatting is too slow and degrades language modeling.
Approach
Train the LLM with a streaming EOS prediction loss on frame tokens (EOS never enters context) alongside standard language modeling loss, so the model learns when to respond or stay silent; generate streaming dialogue data by having an LLM answer templated queries over offline annotation timelines; and speed up inference with a continuous KV cache and parallel encoding/decoding.
Key results
- VideoLLM-online-7B runs at over 10 FPS on a single A100 for 5-minute Ego4D narration streams with under 20 GB memory (18.2G, 13.5 FPS in ablation vs 1.5 FPS for interleaved).
- Streaming dialogue reaches LM-PPL 2.43, TimeDiff 2.32 s and Fluency 42.6%, outperforming interleaved (TimeDiff 6.47) and per-frame streaming baselines on the Ego4D narration validation set.
- On COIN, VideoLLM-online achieves 59.8 step recognition, 92.1 task recognition, 48.1 next forecasting and 52.9 procedure forecasting+goal top-1 accuracy, best among end-to-end models; on Ego4D LTA it reports 0.697 verb / 0.698 noun / 0.897 action ED@20.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace interleaved dialogue training with streaming EOS prediction (ablation) | VideoLLM-online · Ego4D Narration Stream validation | TimeDiff: 6.47 -> 2.32 | Helped |
| replace per-frame dialogue training with streaming EOS prediction (ablation) | VideoLLM-online · Ego4D Narration Stream validation | LM-PPL: 3.34 -> 2.43 | Helped |
| replace interleaved dialogue training with streaming EOS prediction (ablation) | VideoLLM-online · Ego4D Narration Stream validation | Fluency: 11.1% -> 42.6% | Helped |
| replace per-frame dialogue with streaming EOS prediction (ablation) | VideoLLM-online · Ego4D Narration Stream validation | FPS: 7.5 -> 13.5 | Helped |
| replace per-frame dialogue with streaming EOS prediction (ablation) | VideoLLM-online · Ego4D Narration Stream validation | Memory: 24.9G -> 18.2G | Helped |
| replace standard CE with OHEM streaming loss (ablation) | VideoLLM-online · Ego4D Narration Stream validation | LM-PPL: 2.43 -> 2.53 | Hurt |
| replace standard CE with Focal Loss streaming loss (ablation) | VideoLLM-online · Ego4D Narration Stream validation | LM-PPL: 2.43 -> 2.59 | Hurt |
| set streaming loss weight τ = 0.5 instead of τ = 1.0 (ablation) | VideoLLM-online · Ego4D Narration Stream validation | LM-PPL: 2.43 -> 2.44 | No clear effect |
| set streaming loss weight τ = 2.0 instead of τ = 1.0 (ablation) | VideoLLM-online · Ego4D Narration Stream validation | TimeDiff: 2.32 -> 2.31 | No clear effect |
| set streaming loss weight τ = 3.0 instead of τ = 1.0 (ablation) | VideoLLM-online · Ego4D Narration Stream validation | LM-PPL: 2.43 -> 2.47 | Hurt |
| fine-tune VideoLLM-online for COIN recognition and forecasting | VideoLLM-online · COIN benchmarks | Top-1 accuracy | Helped |
| use VideoLLM-online instead of VideoLLM end-to-end model | Ego4D LTA · Ego4D LTA, future 20 actions | ED@Z=20 (Action): 0.921 -> 0.897 | Helped |
Surprising findings
- Per-frame dialogue with GPT-4V produced lengthy outputs at every frame, causing significant delays and making it impractical for streaming video.
- Standard cross-entropy loss for the streaming EOS objective works as well as or better than OHEM and focal loss for handling EOS class imbalance.
Limitations
- Relies on a 1-token-per-frame setup, which limits spatial detail and zero-shot generality.
- Dialogue data is synthesized by an LLM from offline annotations, so responses may be hallucinated or templated.
- Context window of 4096 limits video length to about half an hour in the reported setup.
Experimental setup
Datasets: Ego4D narrations COIN Ego4D GoalStep Ego4D LTA
Benchmarks: COIN Ego4D LTA Ego4D Narration Stream
Metrics: LM-PPL TimeDiff Fluency Top-1 accuracy edit distance FPS memory
Baselines: GPT-4V Interleaved dialogue Per-frame dialogue VideoLLM Palm AntGPT TimeSformer VideoTaskGraph
Builds on: LLaVA LoRA CLIP VideoLLM EgoVLP
Closest papers by meaning
- LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale (CVPR 2025 · similarity 0.90)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.90)
- Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge (ICLR 2025 · similarity 0.90)
- VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation (NeurIPS 2024 · similarity 0.90)
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.89)
- LiveStar: Live Streaming Assistant for Real-World Online Video Understanding (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.