Research map / Video / Video-language understanding
StreamForest: Efficient Online Video Understanding with Persistent Event Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StreamForest, a streaming video MLLM with a Persistent Event Memory Forest that merges event nodes via penalties, plus a Fine-grained Spatiotemporal Window, trained with the OnlineIT dataset and evaluated on the new ODV-Bench driving benchmark.
Problem
Streaming video understanding with MLLMs is limited by the storage cost of historical visual features and weak real-time spatiotemporal reasoning, especially for tasks like autonomous driving.
Approach
Organize streaming frames into event-level tree structures and consolidate them by merging the lowest-penalty adjacent event nodes, where the penalty combines token similarity, merge count, and temporal distance; pair this with a short-term fine-grained spatiotemporal window for current-scene perception.
Key results
- StreamForest reaches 77.3% on StreamingBench, 60.5% on OVBench, and 55.6% on OVO-Bench under the default 8192-token budget.
- With StreamForest (FT-drive), ODV-Bench overall accuracy is 71.2%, versus 59.9% without driving fine-tuning and 55.6% for Qwen2.5-VL.
- Under a 1024-token budget, the model retains 96.8% of its average accuracy across eight benchmarks relative to the default setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune StreamForest on OnlineIT-drive | StreamForest · ODV-Bench | accuracy (Overall): 59.9 -> 71.2 | Helped |
| replace PEMF with FIFO (ablation) | StreamForest memory · MLVU | accuracy (M-Avg): 56.7% vs. 70.0% | Hurt |
| replace PEMF with uniform sampling (ablation) | StreamForest memory · OVBench | accuracy (Avg): 58.2% vs. 60.5% | Hurt |
| replace PEMF with Similarity Merge (ablation) | StreamForest memory · OVO-Bench | accuracy (Overall): +2.2% | Hurt |
| replace PEMF with Pyramid Memory Bank (ablation) | StreamForest memory · MLVU | accuracy (M-Avg): 68.2% vs. 70.0 | Hurt |
| remove both FSTW and PEMF (ablation) | StreamForest · MLVU | accuracy (M-Avg): 51.8 vs. 70.0 | Hurt |
| add FSTW to a model without FSTW (ablation) | StreamForest without FSTW · OVBench | accuracy (Avg): 59.1 -> 60.5 | Helped |
| add PEMF to a model without PEMF (ablation) | StreamForest without PEMF · MLVU | accuracy (M-Avg): 56.6 -> 70.0 | Helped |
| replace event-based node construction with frame-based construction (ablation) | StreamForest · OVO-Bench | accuracy (Overall): 52.6 vs. 55.6 | Hurt |
| add OnlineIT-general to the base training data (ablation) | base model · OVBench | accuracy (Avg): 53.9 -> 60.5 | Helped |
| add OnlineIT-drive after OnlineIT-general (ablation) | base + OnlineIT-general · ODV-Bench | accuracy (Avg): 59.9 -> 71.2 | Helped |
| add OnlineIT-drive after OnlineIT-general (ablation) | base + OnlineIT-general · OVO-Bench | accuracy (Overall): 55.6 -> 55.6 | No clear effect |
Surprising findings
- Uniform sampling and first-in-first-out memory perform close to PEMF on OVBench, indicating that the OVBench gain over baselines is small (60.5 vs 60.3 for Similarity Merge).
Limitations
- Visual token budget and penalty weights (0.4/0.4/0.2) are hand-set hyperparameters; the paper does not show sensitivity analysis beyond the reported ablations.
- ODV-Bench is multiple-choice only and covers driving scenarios; generalization beyond these is not established.
- Some ablation gains are small on OVBench (e.g., PEMF 60.5 vs Similarity Merge 60.3).
Experimental setup
Datasets: OnlineIT-general OnlineIT-drive VideoChat-Flash data LLaVA-Video LLaVA-OneVision image data RefCOCO Visual Genome Charades-STA ActivityNet-Captions TT100k Road-waymo MM-AU
Benchmarks: ODV-Bench StreamingBench OVBench OVO-Bench VideoMME MLVU MVBench PerceptionTest
Metrics: accuracy average accuracy relative retention under token compression latency memory FLOPs
Baselines: MiniCPM-V2.6 LongVA LLaVA-OneVision InternVL2.5 VideoChat-Flash Qwen2.5-VL Flash-VStream Dispider VideoChat-Online Similarity Merge Pyramid Memory Bank
Builds on: ToMe VideoChat-Flash LLaVA-OneVision
Closest papers by meaning
- What Should a Streaming Video Model Remember? (NeurIPS 2026 · similarity 0.90)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.89)
- StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering (NeurIPS 2026 · similarity 0.89)
- CoRDS: Coreset-Based Representative and Diverse Selection for Streaming Video Understanding (NeurIPS 2026 · similarity 0.89)
- LiveStar: Live Streaming Assistant for Real-World Online Video Understanding (NeurIPS 2025 · similarity 0.89)
- FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.