Research map / Video / Video-language understanding
VideoLucy: Deep Memory Backtracking for Long Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoLucy, an agent-based framework with a hierarchical memory (coarse, fine, ultra-fine clip captions) and iterative question-guided backtracking, plus the EgoMem benchmark for long video QA.
Problem
Agent-based long video understanding systems caption individual frames and use sparse sampling, losing temporal context and crucial details in long videos.
Approach
A hierarchical memory of progressively finer clip-level captions (coarse to ultra-fine) is explored by agents in an iterative loop: the localization agent picks the most relevant period, the instruction agent writes a caption prompt, the captioning agent re-describes it at deeper granularity, and the answering agent decides when to stop.
Key results
- Achieves 58.8% overall accuracy on LVBench, about 5.5% above the previous best leaderboard method.
- Reaches 72.5% average on Video-MME (w/o subs), 8.5% above the previous best agent system MemVid.
- Scores 76.1 M-Avg on MLVU, above VideoChat-Flash-7B (74.7) and other listed models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add question-guided deeper memory access, including fine and ultra-fine memories (ablation) | VideoLucy · Video-MME (Long) | accuracy | Helped |
| Add frame-level ultra-fine memory access (ablation) | VideoLucy · Video-MME (Long) | accuracy | Helped |
| Set the maximum number of backtracking iterations to 5 (ablation) | VideoLucy · Video-MME (Long) | accuracy | Helped |
Surprising findings
- Reports that existing open-source MLLMs score only slightly above random guessing on EgoMem.
Limitations
- Iterative loop with multiple agent calls increases inference cost and latency; maximum iteration cap needed to avoid timeouts.
- EgoMem is small (42 videos, 504 QAs) and derived from EgoLife first-person recordings.
Experimental setup
Datasets: MLVU Video-MME LVBench EgoLife
Benchmarks: Video-MME LVBench MLVU EgoMem Needle-in-A-Video-Haystack
Metrics: accuracy Shannon entropy relevance score
Baselines: VideoAgent VideoTree DrVideo MemVid Video-RAG-7B VCA GPT-4o Gemini 1.5 pro Qwen2.5-VL-72B VideoChat-Flash-7B
Builds on: VideoAgent VideoTree DrVideo
Closest papers by meaning
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.89)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.89)
- MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (CVPR 2024 · similarity 0.88)
- $\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation (ICML 2025 · similarity 0.88)
- AdaCM^2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction (CVPR 2025 · similarity 0.88)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.