Research map / Video / Video-language understanding
Streaming Long Video Understanding with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoStreaming, a VLLM that streams long-video encoding with a propagated memory and selects question-related clip memories, using a constant token budget for LLM input.
Problem
Long video understanding with LLMs is bottlenecked by the huge number of visual tokens; prior compression methods either drop temporal information via sparse sampling or lose spatial detail, and memory-bank methods need explicit timestamps to recall history.
Approach
Segment a long video into clips and encode each sequentially with a small LM (Phi-2) that takes the previous clip's condensed memory as input, yielding a fixed-length memory; store all clip memories and use a clip-indicator similarity with a question indicator plus Gumbel-Topk to select a constant number of relevant memories for the LLM.
Key results
- Achieves 44.1% fullset accuracy on EgoSchema zero-shot, outperforming LangRepo 8x7B (41.2%) and comparable to finetuned MC-ViT-L (44.4%).
- On Next-QA validation, reaches 66.2% overall zero-shot accuracy, vs 60.9% for LangRepo 8x7B.
- On MovieNet-QA, uses 256 tokens and 5.32 s latency per question vs 5477 tokens and ~10.5 s for LLaMA-VID without subtitles.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add historical memory to streaming encoding (ablation) | VideoStreaming · MovieChat-1K global mode | accuracy: 43.8 -> 90.4 | Helped |
| Add historical memory to streaming encoding (ablation) | VideoStreaming · EgoSchema fullset | accuracy: 38.4 -> 44.1 | Helped |
| Add adaptive memory selection (ablation) | VideoStreaming · MovieChat-1K breakpoint mode | accuracy: 23.0 -> 54.9 | Helped |
| Add adaptive memory selection (ablation) | VideoStreaming · EgoSchema fullset | accuracy: 37.3 -> 44.1 | Helped |
| Use 24 rather than 16 encoder layers (ablation) | VideoStreaming streaming encoder · EgoSchema fullset | accuracy: 44.1 -> 43.8 | Hurt |
| Use 32 rather than 16 encoder layers (ablation) | VideoStreaming streaming encoder · EgoSchema fullset | accuracy: 44.1 -> 41.3 | Hurt |
| Use 32 rather than 16 encoder layers (ablation) | VideoStreaming streaming encoder · Next-GQA | Acc@GQA: 17.8 -> 15.6 | Hurt |
| Increase summarization tokens P from 1 to 4 with V=4 (ablation) | VideoStreaming · EgoSchema fullset | accuracy: 32.1 -> 44.1 | Helped |
| Increase selected timestamps V from 4 to 8 with P=4 (ablation) | VideoStreaming · EgoSchema fullset | accuracy: 44.1 -> 44.9 | Helped |
| Use clip and memory timestamps in the prompt instead of no time prompt (ablation) | VideoStreaming · MovieChat-1K breakpoint mode | accuracy: 19.8 -> 54.9 | Helped |
| Use cosine similarity instead of dot product for memory selection (ablation) | VideoStreaming · Next-GQA | Acc@GQA: 9.3 -> 17.8 | Helped |
| Use cosine similarity instead of dot product for memory selection (ablation) | VideoStreaming · EgoSchema fullset | accuracy: 34.5 -> 44.1 | Helped |
Surprising findings
- Using fewer Phi-2 layers (16 of 32) gives better results than the full depth, attributed to the next-token-prediction feature space of final layers.
- Dot-product similarity for memory selection causes numerical instability and training overflow, so cosine similarity is used.
Limitations
- Uniform clip segmentation may waste tokens on low-information segments and under-allocate tokens to dynamic segments.
- Long-video QA training relies on noisy or pseudo temporal labels and synthetic concatenated videos.
Experimental setup
Datasets: CC3M WebVid 2.5M LLaVA instruction data Panda-70M movie QA (MovieChat-style) Next-QA EgoSchema MovieChat-1K MovieNet-QA VideoChatGPT benchmark Next-GQA
Benchmarks: VideoChatGPT benchmark EgoSchema Next-QA Next-GQA MovieChat-1K MovieNet-QA
Metrics: accuracy IoP IoU Acc@GQA GPT score (CI, DO, CU, TU, CO) latency tokens
Baselines: LLaMA-VID MovieLLM LangRepo LLoVi Vamos MC-ViT SeViLA MovieChat PLLaVA LongVLM
Builds on: LLaVA LLaMA-VID Phi-2 Vicuna CLIP
Closest papers by meaning
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.93)
- $\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation (ICML 2025 · similarity 0.92)
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.92)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.91)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.91)
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.