Research map / Video / Video-language understanding
Streaming Video Instruction Tuning
Jiaer Xia, Peixian Chen, Mengdan Zhang, Xing Sun, Kaiyang Zhou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use focal loss with frequency weights on state tokens instead of plain cross-entropy (ablation) | Streamo-3B (Qwen2.5-VL-3B) · OVO-Bench Forward Active, SSR | accuracy: 20.99 -> 50.72 | Improved |
| use focal loss instead of plain cross-entropy (ablation) | Streamo-3B (Qwen2.5-VL-3B) · OVO-Bench Forward Active, REC | accuracy: 6.45 -> 27.94 | Improved |
| use focal loss instead of plain cross-entropy (ablation) | InternVL3-2B · OVO-Bench Forward Active, REC | accuracy: 9.46 -> 29.23 | Improved |
| use focal loss instead of fixed class weights (ablation) | Streamo-3B (Qwen2.5-VL-3B) · OVO-Bench Forward Active, SSR | accuracy: 41.02 -> 50.72 | Improved |
| use fixed class weights instead of plain cross-entropy (ablation) | Qwen2.5-VL-3B · OVO-Bench Forward Active, CRR | accuracy: 41.67 -> 49.17 | Improved |
| train on Streamo-Instruct-465K instead of ET-Instruct-164K (ablation) | Streamo framework 3B · OVO-Bench overall | overall avg accuracy: 40.54 -> 52.33 | Improved |
| add offline LLaVA-Video data to ET-Instruct training (ablation) | ET-Instruct-3B · OVO-Bench Real-Time vs Forward Active | accuracy: Real-Time avg 46.47 -> 57.78; Forward avg 46.62 -> 40.77 | Mixed |
| evaluate at 2 fps without retraining (trained at 1 fps) | Streamo-7B · OVO-Bench overall | overall avg accuracy: 55.61 -> 57.86 | Improved |
| convert Qwen2.5-VL-7B to streaming with Streamo training | Qwen2.5-VL-7B · OVO-Bench Real-Time | accuracy: 58.8 -> 66.0 | Improved |
| convert Qwen2.5-VL-7B to streaming with Streamo training | Qwen2.5-VL-7B · TempCompass | accuracy: 71.7 -> 71.8 | No clear effect |
| convert Qwen2.5-VL-3B to streaming with Streamo training | Qwen2.5-VL-3B · offline benchmarks average | average accuracy: 56.6 -> 59.2 | Improved |
| train with ET-Instruct-164K only | Qwen2.5-VL-3B · offline benchmarks average | average accuracy: 56.6 -> 51.5 | Hurt |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 2 of 2 on most shared results.
- Dispider-7B — Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction (CVPR 2025) · better on 2 of 2 shared results
- StreamingVLM-7B — StreamingVLM: Real-Time Understanding for Infinite Video Streams (ICLR 2026) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge (ICLR 2025 · similarity 0.90)
- StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant (NeurIPS 2025 · similarity 0.89)
- LiveStar: Live Streaming Assistant for Real-World Online Video Understanding (NeurIPS 2025 · similarity 0.89)
- StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering (NeurIPS 2026 · similarity 0.89)
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.88)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
