Research map / Video / Video-language understanding
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MovieChat, a training-light long-video MLLM that uses an Atkinson-Shiffrin-inspired short-term/long-term token memory with ToMe-style merging, and releases the MovieChat-1K benchmark.
Problem
Video LLM systems handle only a few frames; long videos (>10K frames) pose computation, memory, and long-range temporal connection challenges.
Approach
Process frames with a sliding window into a fixed-length short-term token buffer (FIFO); when full, consolidate tokens by iteratively merging the most similar adjacent frames (ToMe-style) into a sparse long-term memory, with global and breakpoint inference modes.
Key results
- On MovieChat-1K global mode, MovieChat reaches 62.3% accuracy and 3.23 score versus 57.8% / 3.00 for Video Chat (32 frames).
- Memory mechanism ablation on MovieChat-1K global mode raises accuracy from 51.4 (w/o memory) to 67.8 (base).
- On MSVD-QA, MovieChat achieves 75.2% accuracy versus 64.9% for Video-ChatGPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the memory mechanism instead of removing it (ablation) | MovieChat · MovieChat-1K global mode | Accuracy: 51.4 -> 67.8 | Helped |
| Use the memory mechanism instead of removing it (ablation) | MovieChat · MovieChat-1K global mode | Score: 3.10 -> 3.81 | Helped |
| Use the memory mechanism instead of removing it (ablation) | MovieChat · MovieChat-1K breakpoint mode | Accuracy: 38.2 -> 50.4 | Helped |
| Use the memory mechanism instead of removing it (ablation) | MovieChat · MovieChat-1K breakpoint mode | Score: 2.31 -> 2.96 | Helped |
| Use merged tokens for short-term initialization rather than last-token or uniform sampling (ablation) | MovieChat · MovieChat-1K | accuracy | Helped |
| Vary long-term memory buffer length (ablation) | MovieChat · MovieChat-1K | accuracy | Mixed |
| Vary short-term memory buffer length (ablation) | MovieChat · MovieChat-1K | accuracy | Mixed |
Surprising findings
- On MSRVTT-QA, MovieChat's score (2.6) is lower than Video-ChatGPT's accuracy-based ranking would suggest, and Video Chat scores 2.5 on MSRVTT-QA versus MovieChat's 2.6 with an accuracy of 52.7 vs 45.0, showing mixed metric agreement.
Limitations
- Performance is bounded by the pretrained short-video understanding model.
- Provides only rough estimates of event duration proportions, lacking temporal precision.
- Early-stage prototype; evaluation relies on LLM-assisted judging with manual filtering of inconsistent results.
Experimental setup
Datasets: MovieChat-1K MSVD-QA MSRVTT-QA ActivityNet-QA
Benchmarks: MovieChat-1K MSVD-QA MSRVTT-QA ActivityNet-QA
Metrics: accuracy score (0-5 scale) correctness of information detail orientation contextual understanding temporal understanding consistency
Baselines: Video Chat Video LLaMA Video-ChatGPT LLaMA Adapter FrozenBiLM
Builds on: Video-ChatGPT ToMe BLIP-2 Atkinson-Shiffrin memory model Video-LLaMA
Closest papers by meaning
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (CVPR 2024 · similarity 0.91)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.90)
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.89)
- MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding (CVPR 2024 · similarity 0.89)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.88)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.