Research map / Video / Video-language understanding

FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding

NeurIPS 2026 · Video › Video-language understanding · approach: Long video understanding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FreshMem, a training-free, brain-inspired frequency-space hybrid memory (multi-scale frequency memory plus space thumbnail memory) that plugs into video MLLMs for streaming video understanding, with optional component-wise fine-tuning reaching 82.31% on StreamingBench.

Problem

Streaming video understanding with MLLMs requires balancing short-term detail fidelity with long-term coherence under causal constraints; existing KV-cache pruning, token condensation, and memory banks lose fine-grained detail or fragment context.

Approach

Overflowed frames are compressed into incremental DFT frequency coefficients (with decay and top-k norm-based residual tokens) for a global gist, while cosine-similarity episode segmentation with adaptive sampling and centroid-based merging yields space thumbnails; both are concatenated with a sliding window and fed to the LLM.

Key results

• Training-free FreshMem on Qwen2-VL-7B improves StreamingBench to 74.20% (+5.20), OV-Bench to 50.82% (+4.52), and OVO-Bench to 54.53% (+2.34).
• With LLM fine-tuning, FreshMem reaches 82.31% on StreamingBench, a 13.31-point gain over the Qwen2-VL-7B baseline (69.00%) and above StreamForest (77.26%).
• On offline benchmarks, FreshMem raises Qwen2VL-7B on MLVU from 54.9 to 57.3 and on MVBench from 59.1 to 63.8.

Surprising findings

• The paper reports FreshMem's training-free version exceeds several fully fine-tuned streaming methods.
• Inference overhead is modest: latency rises from 81 to 95 minutes and GPU memory from 17GB to 20GB on OV-Bench AVA.

Limitations

• Gains on some subtasks are small, and the checklist-level limitation discussion is referenced but not clearly provided.
• Fine-tuned results rely on the StreamForest end-to-end training recipe, and the training data is unspecified in the text.

Experimental setup

Datasets: StreamingBench OV-Bench OVO-Bench MLVU MVBench

Benchmarks: StreamingBench OV-Bench OVO-Bench MLVU MVBench

Metrics: accuracy

Baselines: Qwen2-VL-7B InternVL2-2B StreamForest Dispider Flash-VStream LiveVLM ReKV MovieChat MA-LMM VideoLLaMB

Builds on: Qwen2-VL InternVL2 StreamForest

Closest papers by meaning

• Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.89)
• StreamForest: Efficient Online Video Understanding with Persistent Event Memory (NeurIPS 2025 · similarity 0.88)
• What Should a Streaming Video Model Remember? (NeurIPS 2026 · similarity 0.87)
• Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion (NeurIPS 2026 · similarity 0.87)
• ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.87)
• VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion (NeurIPS 2026 · similarity 0.87)