Research map / Video / Video-language understanding
StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StreamBridge, a plug-and-play framework that converts offline Video-LLMs into streaming multi-turn, proactive assistants via a memory buffer with round-decayed compression and a decoupled activation model, plus the Stream-IT dataset.
Problem
Offline Video-LLMs process entire pre-recorded videos at once and cannot handle multi-turn real-time interaction or proactive responses in streaming video scenarios.
Approach
Keep the offline Video-LLM frozen in architecture and add (1) a memory buffer with round-decayed token merging that compresses older visual rounds first, and (2) a separate lightweight activation model (LLaVA-OV-0.5B with a score head and activation token) that decides per frame when the main model should respond.
Key results
- Qwen2-VL-7B with StreamBridge and Stream-IT reaches 71.30 average on OVO-Bench real-time and 77.04 on Streaming-Bench, exceeding GPT-4o and Gemini 1.5 Pro in the paper's tables.
- Oryx-1.5-7B with Stream-IT improves OVO-Bench real-time average by +11.92 (59.25 to 71.17) under the multi-turn streaming setup.
- Oryx-1.5-7B with StreamBridge reaches 65.5 on VideoMME (up 6.7) while retaining general video performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune under StreamBridge on Stream-IT | Qwen2-VL-7B · OVO-Bench Real-Time | AVG. accuracy: 63.35 -> 71.30 | Helped |
| Fine-tune under StreamBridge on Stream-IT | Qwen2-VL-7B · Streaming-Bench Real-Time | AVG. accuracy: 72.01 -> 77.04 | Helped |
| Fine-tune under StreamBridge on Stream-IT | Oryx-1.5-7B · OVO-Bench Real-Time | AVG. accuracy: +11.92 | Helped |
| Fine-tune under StreamBridge on Stream-IT | LLaVA-OV-7B · OVO-Bench Real-Time | AVG. accuracy: 61.64 -> 69.93 | Helped |
| Add StreamBridge streaming setup without Stream-IT fine-tuning | LLaVA-OV-7B · OVO-Bench Real-Time | AVG. accuracy: 64.02 -> 61.64 | Hurt |
| Add StreamBridge streaming setup without Stream-IT fine-tuning | Qwen2-VL-7B · OVO-Bench Real-Time | AVG. accuracy: 55.98 -> 63.35 | Helped |
| Add StreamBridge and fine-tune on Stream-IT | Oryx-1.5-7B · Video understanding benchmarks | accuracy: 58.8 -> 65.5 | Helped |
| Use StreamBridge and fine-tune on Stream-IT | LLaVA-OV-7B · LongVideoBench | accuracy: 56.3 -> 54.3 | Hurt |
| Replace round-decayed compression with truncation (ablation) | StreamBridge with Qwen2-VL-7B · OVO-Bench | AVG. accuracy: 71.30 -> 68.88 | Hurt |
| Replace round-decayed compression with round-uniform compression (ablation) | StreamBridge with Qwen2-VL-7B · Streaming-Bench | AVG. accuracy: 77.04 -> 74.18 | Hurt |
| Remove StreamingQA-120K from Stream-IT while retaining LLaVA-178K (ablation) | StreamBridge with Qwen2-VL-7B · OVO-Bench | AVG. accuracy: 71.30 -> 67.67 | Hurt |
| Remove StreamingQA-120K from Stream-IT while retaining LLaVA-178K (ablation) | StreamBridge with Qwen2-VL-7B · Streaming-Bench | AVG. accuracy: 77.04 -> 72.42 | Hurt |
Surprising findings
- Without Stream-IT, LLaVA-OV-7B drops under the streaming setup (OVO-Bench 64.02 to 61.64), while Qwen2-VL improves, which the authors attribute to differences in interleaved pretraining data.
- Training on LLaVA-178K alone causes a marked drop on OVO-Bench and Streaming-Bench, showing that offline data does not transfer to multi-turn streaming.
Limitations
- Stream-IT relies partly on synthetic QA generation and clip concatenation, which may cause domain shift from real continuous streams.
- Evaluated only frame-by-frame at low sampling rates (1 FPS); denser or audio-visual streaming is not addressed.
Experimental setup
Datasets: Stream-IT StreamingQA-120K LLaVA-178K VCG-Plus ShareGPT4Video ActivityNet Captions Shot2Story YouCook2 COIN FineAction HACS Charades
Benchmarks: OVO-Bench Streaming-Bench MVBench PerceptionTest TempCompass EgoSchema LongVideoBench MLVU VideoMME ET-Bench
Metrics: accuracy F1 averaged over IoU thresholds sentence-level similarity
Baselines: GPT-4o Gemini 1.5 Pro Qwen2-VL-72B LLaVA-Video-7B InternVL-V2-8B Flash-VStream VideoLLM-Online Dispider
Builds on: LLaVA-OneVision Qwen2-VL Oryx
Closest papers by meaning
- Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge (ICLR 2025 · similarity 0.91)
- LiveStar: Live Streaming Assistant for Real-World Online Video Understanding (NeurIPS 2025 · similarity 0.90)
- StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering (NeurIPS 2026 · similarity 0.88)
- VideoLLM-online: Online Video Large Language Model for Streaming Video (CVPR 2024 · similarity 0.88)
- StreamForest: Efficient Online Video Understanding with Persistent Event Memory (NeurIPS 2025 · similarity 0.88)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.