Research map / Video / Video-language understanding

What Should a Streaming Video Model Remember?

NeurIPS 2026 · Video › Video-language understanding · approach: Long video understanding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SelectStream, a selective latent-memory framework that writes projected VLM visual embeddings into a fixed-capacity query-conditioned graph and injects calibrated latent evidence tokens into a frozen VLM for streaming video QA.

Problem

Streaming video models must answer queries at any time under fixed memory and compute budgets, but indiscriminate history injection can dilute current-scene perception and recent-window baselines remain strong.

Approach

Treat streaming memory as budgeted online latent evidence allocation: surprise-driven adaptive segmentation decides when to write, priority-aware graph consolidation decides what to keep, and query-conditioned graph attention decides what to read, with only a small number of calibrated latent evidence tokens exposed to a frozen VLM.

Key results

• Reaches 82.67% on StreamingBench with SelectStream-Qwen3-VL-8B and 81.42% with Qwen2.5-VL-7B, versus 80.59% for Qwen3-VL-8B + 4f.
• On OVO-Bench, RT/BT average rises from 65.13% to 70.95% for Qwen2.5-VL-7B; BT Avg improves from 51.90% to 61.05%.
• Offline average reaches 74.4% on VideoMME, MLVU, and MVBench with Qwen3-VL-8B, +1.7 over the frozen backbone.

Surprising findings

• Retrieved-frame replay is outperformed by calibrated latent evidence tokens, while raw retrieved states without calibration lose 2.5-2.8 points.
• Strong recent-window baselines can match or exceed memory-heavy streaming systems, suggesting indiscriminate history injection dilutes current perception.

Limitations

• Reported main results rely on a Streamo-derived training set and baselines whose budgets are not exactly matched.
• Proposition 2 assumptions (retrieval recall, logit bounds) are not verified architecturally and are only estimated empirically.

Experimental setup

Datasets: Streamo-Instruct-465K

Benchmarks: StreamingBench OVO-Bench VideoMME MLVU MVBench

Metrics: accuracy RT/BT Avg Recall@M T-Overlap TTFT peak GPU memory

Baselines: SimpleStream Streamo StreamForest Flash-VStream HERMES ThinkStream Qwen2.5-VL-7B + 4f Qwen3-VL-8B + 4f

Builds on: Streamo-Instruct-465K

Closest papers by meaning

• CoRDS: Coreset-Based Representative and Diverse Selection for Streaming Video Understanding (NeurIPS 2026 · similarity 0.91)
• StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering (NeurIPS 2026 · similarity 0.91)
• Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.90)
• StreamForest: Efficient Online Video Understanding with Persistent Event Memory (NeurIPS 2025 · similarity 0.90)
• ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.88)
• LiveStar: Live Streaming Assistant for Real-World Online Video Understanding (NeurIPS 2025 · similarity 0.88)