Research map / Video / Video-language understanding
Memory Consolidation Enables Long-Context Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MC-ViT, which fine-tunes pretrained video transformers to attend to non-parametrically consolidated (k-means, coreset, or random) memories of past segment activations, achieving state-of-the-art long-video QA and Diving48 results.
Problem
Standard video transformers have quadratic attention cost and are limited to short temporal contexts, making long-video understanding expensive.
Approach
Process video in segments and cross-attend to a finite memory built non-parametrically by consolidating (k-means clustering, coreset, or random selection) past segment activations at each layer, letting pretrained ViViT weights be re-purposed with only light fine-tuning.
Key results
- MC-ViT-L reaches 62.6 on EgoSchema subset and 44.4 on full EgoSchema, and 48.1 on Perception Test, outperforming models with up to 5B parameters.
- MC-ViT-B achieves 89.7 top-1 on Diving48 without extra spatial crops or bounding boxes, and MC-ViT-L achieves 91.0, exceeding AIM ViT-L (90.6).
- MC-ViT outperforms joint space-time attention with 10x less memory and FLOPS, and 16x memory compression with k-means (128 memories per segment) surpasses all baselines on Diving48.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| consolidate past activations with k-means rather than use uncompressed past activations (ablation) | MC-ViT on Diving48 · Diving48 | accuracy | Helped |
| select memories randomly rather than use uncompressed past activations (ablation) | MC-ViT on Diving48 · Diving48 | accuracy | Helped |
| select a coreset of representative past activations (ablation) | MC-ViT on Diving48 · Diving48 | accuracy | Helped |
| use globally selected memory tokens instead of retaining only the last segments (ablation) | MC-ViT-B · Diving48 | accuracy: 81.5 -> 84.8 | Helped |
| use globally selected memory tokens instead of retaining only the last segments (ablation) | MC-ViT-B · Diving48 | accuracy: 87.8 -> 87.8 | No clear effect |
| train MC-ViT with longer video contexts (ablation) | MC-ViT on Diving48 · Diving48 | accuracy | Helped |
| add cross-segment memory attention rather than use memory-less streaming (ablation) | MC-ViT on Diving48 · Diving48 | accuracy | Helped |
| increase video length for joint space-time attention (ablation) | joint space-time attention on Diving48 · Diving48 | accuracy | Mixed |
| retain the last 5 segments instead of a smaller memory (ablation) | MC-ViT-B · Diving48 | accuracy: 81.5 -> 87.8 | Helped |
| replace joint space-time attention with MC-ViT memory consolidation (ablation) | MC-ViT on Diving48 · Diving48 | FLOPS: 10× less | Helped |
| replace joint space-time attention with MC-ViT for training (ablation) | MC-ViT on Diving48 · Diving48 | training FLOPS: 3.2× fewer | Helped |
Surprising findings
- Even random memory selection outperforms all baselines at 4x compression with 512 memories on Diving48.
- SeViLA remains stronger than MC-ViT on Next-QA (73.8 vs 65.0), which the authors attribute to language-only reasoning in that benchmark.
Limitations
- Performance on Next-QA lags SeViLA, which relies on large LLM decoders and language priors.
- Proprietary model comparisons are based on zero-shot API evaluation with unknown model details and possible data contamination concerns.
- Only simple non-parametric consolidation schemes were explored; memory selection/quality tradeoffs beyond the three tested are not studied.
Experimental setup
Datasets: Diving48 EgoSchema Ego4D HowTo100M Next-QA Perception Test ALIGN LTIP JFT VTP
Benchmarks: Diving48 EgoSchema Perception Test Next-QA
Metrics: top-1 accuracy accuracy (percentage correct)
Baselines: ViViT joint space-time attention ST-ViT MeMViT ImageViT ShortViViT LongViViT SeViLA Flamingo InternVideo GPT-4V Gemini Ultra Bard + PALI
Builds on: ViViT Transformer-XL CLIP MeMViT Perceiver resampler
Closest papers by meaning
- $\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation (ICML 2025 · similarity 0.88)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.87)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.87)
- A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames (CVPR 2024 · similarity 0.87)
- MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (CVPR 2024 · similarity 0.87)
- Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.