Research map / Video / Video-language understanding
StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StreamOV, a streaming omni-video framework with evidence-guided long-short term memory and a hidden-state trigger on a frozen Qwen3-Omni-30B-A3B, plus the SOVBench benchmark for multi-turn online audio-visual QA and response triggering.
Problem
Omni-modal video LLMs are built for offline inference; they cannot manage growing audio-visual context over long streams or decide when to respond proactively, and existing benchmarks lack multi-turn online audio-visual evaluation.
Approach
Builds query-agnostic and query-aware multimodal evidence scores to route frames into visual-only, audio-only, and audio-visual-aligned cues, keeps a bounded short-term and long-term memory, and trains a lightweight cross-attention trigger on the frozen MLLM's prefill hidden state to decide Respond vs Wait, avoiding silence tokens and external routers.
Key results
- On SOVBench-T, StreamOV reaches F1 90.5 versus 52.2 for ROMA-7B; on SOVBench-O average accuracy it reaches 81.6 (Audio-Visual Context) vs 73.7 for Qwen3-Omni-30B.
- On StreamingBench Audio-Visual QA, StreamOV averages 68.6 versus 61.0 for Qwen3-Omni-30B; Visual-Only QA average 86.2 vs 77.9.
- On Video-MME (no subtitles), StreamOV reaches 73.5 overall versus 68.5 for Qwen3-Omni-30B.
Surprising findings
- Qwen3-Omni-30B, run offline at 1 FPS, outperforms StreamOV on some SOVBench-O Recall and Real-Time cells, which the authors attribute to its less constrained offline setting.
- Using two hidden states (prefill plus first decode) gives only a marginal F1 gain (90.5 to 90.7) over a single prefill state.
Limitations
- Memory update relies on predefined evidence scores and heuristic temporal refinement rather than adaptive planning.
- Trigger supports only binary Respond/Wait actions; finer actions such as clarification are not handled.
- Trigger becomes less effective when audio-visual information is highly sparse, due to limits of the underlying MLLM.
Experimental setup
Datasets: FineVideo StreamingBench OVO-Bench Video-Holmes Daily-Omni Video-MME
Benchmarks: SOVBench-O SOVBench-T StreamingBench OVO-Bench Video-Holmes Daily-Omni Video-MME
Metrics: accuracy precision recall F1
Baselines: Qwen3-Omni-30B-A3B Qwen2.5-Omni-7B Qwen2.5-VL-7B Qwen3-VL-30B-A3B ROMA StreamForest TimeChat-Online Dispider Flash-VStream VideoLLM-online Gemini 2.5 Flash Gemini 3.0 Flash
Builds on: Qwen3-Omni
Closest papers by meaning
- A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.91)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.91)
- What Should a Streaming Video Model Remember? (NeurIPS 2026 · similarity 0.91)
- OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts (CVPR 2025 · similarity 0.90)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.90)
- LiveStar: Live Streaming Assistant for Real-World Online Video Understanding (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.