Research map / Video / Video-language understanding
$\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free extension of short-context video Q-formers (Video-LLaMA, VideoChat2) with a continuous-time long-term memory based on continuous attention with Gibbs densities and sticky (density-based) sampling, evaluated on long-video QA benchmarks.
Problem
Short-context video-language models cannot process arbitrarily long videos and rely on sparse frame subsampling, losing critical information.
Approach
Adapts the infinite-former continuous attention to video Q-former cross-attention: frames are pooled and represented by a ridge-regression basis, a long-term memory is consolidated by contracting past context with a forgetting factor, and sampling is biased toward high-attention regions ("sticky memories"), using a Gibbs PDF over continuous query-key similarity instead of a Gaussian.
Key results
- On EgoSchema, the infinity-Video LLaMA with sticky LTM (alpha=0.9) reports roughly +6 points over the no-LTM variant (alpha=1).
- On MovieChat-1K, infinity-Video LLaMA (sticky) reaches 72.2% accuracy and 3.88 score versus 67.8% / 3.81 for MovieChat.
- On NeXT-QA, infinity-Video LLaMA (sticky) achieves 41.1 vs 30.7 for Video-LLaMA; infinity-VideoChat2 gains are small (78.1 vs 78.7 for VideoChat2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use sticky rather than uniform memory sampling (ablation) | ∞-Video LLaMA · NeXT-QA | accuracy: 37.5 -> 41.1 | Helped |
| add LTM with sticky memory sampling (ablation) | ∞-Video LLaMA · EgoSchema | accuracy: 40.8 -> 46.8 | Helped |
| use sticky rather than uniform memory sampling (ablation) | ∞-Video LLaMA · EgoSchema | accuracy: 42.6 -> 46.8 | Helped |
| add LTM to VideoChat2 (ablation) | ∞-VideoChat2 · NeXT-QA | accuracy: 78.7 -> 78.1 | Hurt |
| add LTM to VideoChat2 (ablation) | ∞-VideoChat2 · EgoSchema | accuracy: 64.2 -> 64.8 | Helped |
| use sticky rather than uniform memory sampling (ablation) | ∞-VideoChat2 · EgoSchema | accuracy: 64.4 -> 64.8 | Helped |
| use sticky rather than uniform memory sampling (ablation) | ∞-VideoChat2 · VideoMME Long | accuracy: 38.8 -> 38.9 | Helped |
| use sticky rather than uniform memory sampling (ablation) | ∞-Video LLaMA · MovieChat-1K | Accuracy: 66.5 -> 72.2 | Helped |
| use uniform memory sampling rather than sticky sampling (ablation) | ∞-Video LLaMA · MovieChat-1K | Accuracy: 68.0 -> 66.5 | Hurt |
| remove STM and use sticky LTM sampling (ablation) | ∞-Video LLaMA · MovieChat-1K | Accuracy: 72.2 -> 59.2 | Hurt |
| remove STM and use uniform LTM sampling (ablation) | ∞-Video LLaMA · MovieChat-1K | Accuracy: 72.2 -> 62.4 | Hurt |
| remove STM and use sticky LTM sampling (ablation) | ∞-VideoChat2 · MovieChat-1K | Accuracy: 63.9 -> 66.5 | Helped |
Surprising findings
- For VideoChat2 variants, the LTM yields no significant gains on in-domain NeXT-QA, and replacing STM with LTM alone performs best on MovieChat, while for Video-LLaMA LTM-only underperforms.
- The stronger VideoChat2 model underperforms the Video-LLaMA category on open-ended MovieChat metrics despite larger training data.
Limitations
- Gains for VideoChat2 are small or absent on in-domain short-video benchmarks.
- Relies on GPT-3.5 as judge for open-ended evaluation and on similarity matching for multiple-choice answers.
- Multiple-choice evaluation required feeding only the question to the model because of format-following issues.
Experimental setup
Datasets: NeXT-QA EgoSchema Video-MME MovieChat-1K
Benchmarks: NeXT-QA EgoSchema Video-MME MovieChat-1K
Metrics: accuracy score (0-5) contextual understanding detail orientation correctness of information
Baselines: Video-LLaMA MovieChat MovieChat+ VideoChat2 VideoAgent VideoTree LLoVi Video-LLaVA
Builds on: infinite-former Video-LLaMA VideoChat2 continuous attention
Closest papers by meaning
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.92)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.91)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.90)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.90)
- MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding (CVPR 2024 · similarity 0.90)
- AdaCM^2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.