Research map / Video / Video-language understanding
Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StreamChat, a training-free streaming video-LLM with hierarchical memory (short-term, long-term tree, dialogue) and three-thread system scheduling built on LongVA, plus the StreamBench benchmark for multi-round streaming video QA.
Problem
Video-LLMs struggle with long video sequences, multi-turn dialogue, and real-time latency in dynamic streaming scenarios.
Approach
A training-free framework combining optical-flow-based selective frame stacking, a hierarchical memory (k-means clustered long-term tree with captions, random-forgetting short-term buffer, and MiniLM-encoded dialogue memory), and parallel threads for frame encoding, memory formation, and response generation.
Key results
- On StreamBench online setting, the Slow variant reaches 64.7% accuracy and 3.48 score, 8.3% above Video-online (56.4%).
- The Fast variant processes video at 32 FPS with 0.85s request processing delay, versus Flash-VStream's 1 FPS and 4.15s delay.
- On offline benchmarks, StreamChat averages 50.6% accuracy across four datasets, improving LongVA's average by 2.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add long-term memory (ablation) | StreamChat Base · StreamBench LM | accuracy: 45.5 -> 51.7 | Helped |
| Add short-term memory (ablation) | StreamChat Base · StreamBench SM | accuracy: 52.5 -> 55.7 | Helped |
| Add dialogue memory (ablation) | StreamChat Base · StreamBench CI | accuracy: 60.1 -> 64.2 | Helped |
| Add long-term and short-term memory together (ablation) | StreamChat Base · StreamBench average | accuracy: 62.2 -> 63.1 | Helped |
| Increase frame-selection threshold (ablation) | StreamChat · StreamBench | accuracy: 64.0% -> 60.7% | Hurt |
| Increase long-memory chunk length from 15 to 30 (ablation) | StreamChat Base · StreamBench | accuracy: 61.2%→64.0% | Helped |
| Increase long-memory chunk length from 30 to 40 (ablation) | StreamChat Base · StreamBench | accuracy: 64.0%→63.1% | Hurt |
| Increase long-memory group size from 2 to 12 (ablation) | StreamChat Base · StreamBench | accuracy: 62.0%→63.9% | Helped |
| Increase clustering goal from 3 to 10 (ablation) | StreamChat Base · StreamBench | accuracy: 59.4%→64.0% | Helped |
| Use StreamChat Slow instead of Video-online | StreamChat · StreamBench | accuracy: 56.4 -> 64.7 | Helped |
| Use StreamChat Fast instead of Video-online | StreamChat · StreamBench | FPS: 5 -> 32 | Helped |
| Add StreamChat memory and processing to LongVA | LongVA · Offline benchmarks average | accuracy: 48.1 -> 50.6 | Helped |
Surprising findings
- Increasing the optical-flow threshold to speed up processing reduces accuracy (64.0% to 60.7%), so higher throughput trades off against performance.
- Human performance on StreamBench (79.4% accuracy) is only modestly above the best model's 64.7%, while GPT-4o reaches 71.0% with 50 frames.
Limitations
- Retrieval relies on basic cosine-similarity matching, sometimes producing incorrect responses.
- VRAM consumption of the tree-structured memory limits scalability to longer videos (up to 56 GB when clustering goal increases).
- Failures on small or briefly appearing objects and on target action interpretation.
Experimental setup
Datasets: EgoSchema YouTube-8M MSVD-QA MSRVTT-QA ActivityNet-QA NExT-QA MovieChat
Benchmarks: StreamBench MSVD-QA MSRVTT-QA ActivityNet-QA NExT-QA
Metrics: semantic score (0-5) accuracy coherence request processing delay FPS
Baselines: GPT-4o GPT-4o-mini Video-LLaVA LLaMA-VID VILA1.5 InternVL2 LongVA LLaVA-Hound MovieChat FreeVA Video-online Flash-VStream
Builds on: LongVA MovieChat Video-online Flash-VStream MemoryBank
Closest papers by meaning
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.93)
- StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant (NeurIPS 2025 · similarity 0.91)
- VideoLLM-online: Online Video Large Language Model for Streaming Video (CVPR 2024 · similarity 0.90)
- StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering (NeurIPS 2026 · similarity 0.89)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.89)
- Online Video Understanding: OVBench and VideoChat-Online (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.