Research map / Video / Video-language understanding
ReWind: Understanding Long Videos with Instructed Learnable Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReWind, a memory-based VLM for long video understanding that uses a learnable read-perceive-write memory and instruction-guided dynamic frame selection to answer questions and ground events in long videos.
Problem
Existing VLMs struggle with long videos due to quadratic self-attention cost, memory limits, and loss of temporal fidelity when compressing video; current memory modules store information without regard to the instruction.
Approach
A learnable memory bank updated by a read-perceive-write cycle (learnable read queries cross-attend to memory and initialize a per-frame perceiver; learnable write queries compress outputs into memory), plus a second stage that selects instruction-relevant high-resolution frames via attention scoring and density-peaks clustering from a feature buffer.
Key results
- On MovieChat-1K global VQA, ReWind reaches 80.6 accuracy and 4.46 score versus 67.8 and 3.81 for MovieChat, while using about 1/8 of the tokens and 1/4 of the frames.
- On Charades-STA temporal grounding, ReWind achieves 39.3 mIoU and 41.6 R@0.3, versus 31.2 mIoU for VTimeLLM.
- On the VideoChatGPT short-video benchmark, ReWind achieves an average generation score of 2.91, the highest among the compared methods (LLaMA-VID 2.89, Chat-UniVi 2.89).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the memory module (ablation) | ReWind baseline · MovieChat-1K test, global | accuracy: 61.5 -> 76.8 | Helped |
| Add dynamic frame selection to the memory model (ablation) | ReWind · MovieChat-1K test, global | accuracy: 76.8 -> 80.6 | Helped |
| Add the memory module (ablation) | ReWind baseline · VideoChatGPT test | AVG: 2.72 -> 2.73 | No clear effect |
| Add dynamic frame selection to the memory model (ablation) | ReWind · VideoChatGPT test | AVG: 2.73 -> 2.91 | Helped |
| Condition the perceiver on text and read information (ablation) | ReWind perceiver · MovieChat-1K test, global | accuracy: 69.1 -> 76.8 | Helped |
| Increase memory storage from 2 to 4 tokens per frame (ablation) | ReWind without DFS · MovieChat-1K test, global | accuracy: 76.8 -> 79.7 | Helped |
| Increase DFS candidate frames from 64 to 128 (ablation) | ReWind · MovieChat-1K test, global | accuracy: 80.6 -> 80.1 | Hurt |
| Increase final DFS frames from 8 to 16 (ablation) | ReWind · MovieChat-1K test, breakpoint | accuracy: 57.2 -> 55.2 | Mixed |
| Replace DFS with uniform sampling (ablation) | ReWind · MovieChat-1K test, global | accuracy: 80.6 -> 77.5 | Hurt |
| Increase input frames from 512 to 1024 (ablation) | ReWind · MovieChat-1K test | performance | Hurt |
| Increase input frames (ablation) | ReWind · MovieChat-1K test | memory consumption: 29GB | Hurt |
Surprising findings
- Increasing input frames beyond 512 (around 1 fps) degrades performance at 1024 frames, which the authors attribute to deviation from the training regime and token redundancy.
- Text-conditioning and read-conditioning of the perceiver each matter; removing read conditioning drops global accuracy from 76.8 to 69.1 in the ablation.
Limitations
- Hallucinations in generated answers, as shown in the qualitative example.
- Evaluation relies on GPT-3.5 as judge, and the temporal grounding and VQA benchmarks cover only a limited set of tasks.
- Performance drops at higher frame rates (1024 frames), indicating sensitivity to the training regime.
Experimental setup
Datasets: WebVid2.5M Panda70M VideoChatGPT instruction data DiDeMo ActivityNet Captions MovieChat-1K Charades-STA VideoChatGPT test set
Benchmarks: MovieChat-1K Charades-STA VideoChatGPT test set
Metrics: accuracy score CI DO CU TU CO R@0.3 R@0.5 R@0.7
Baselines: Video LLaMA Video-ChatGPT Video Chat MovieChat GroundingGPT TimeChat VTimeLLM MA-LMM LLaMA-VID Chat-UniVi LLaMA Adapter
Builds on: Q-Former MovieChat DPC-KNN SigLIP LoRA
Closest papers by meaning
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.93)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.92)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.91)
- $\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation (ICML 2025 · similarity 0.90)
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.90)
- AdaCM^2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.