Research map / Video / Video-language understanding
Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a zero-shot pipeline that uses LLaVA 1.6 to describe and reason over key frames and Llama 3.1 to merge them with a textual memory chain, plus a new VAD-Score metric for video description quality, evaluated on 10 DREAM-1K videos.
Problem
Large video-language models struggle with fine-grained understanding, generalization to real-world videos, and long-term temporal memory, leading to incomplete descriptions of complex videos.
Approach
Combine an off-the-shelf VLM (LLaVA 1.6) for per-keyframe descriptions and six Q/A reasoning pairs with an LLM (Llama 3.1) that fuses frame descriptions iteratively, using the running textual summary as memory; score outputs with a weighted F1 metric over events, subjects and objects (VAD-Score).
Key results
- Short videos scored VAD 0.692 versus 0.595 on long videos.
- On live action, event F1 0.75 vs GPT-4V 0.75 and Gemini 1.5 Pro 0.77.
- On movie animation and YouTube, event F1 of 0.29 trailed PLLaVA-34B and LLaVA-NeXT-Video.
Surprising findings
- Performance is weaker on long videos than short ones, and the method lags on movie animation and YouTube categories.
Limitations
- Evaluated on only 10 randomly sampled videos.
- Ground truth manually created by the authors, limited scale.
- Underperforms baselines on movie animation and YouTube categories.
Experimental setup
Datasets: DREAM-1K
Benchmarks: DREAM-1K
Metrics: VAD-Score event F1 subject F1 object F1
Baselines: Video-LLaVA MiniGPT-4VT LLaVA-NeXT-Video VideoChat2 PLLaVA-34B GPT-4V Gemini 1.5 Pro Tarsier-7B
Builds on: LLaVA-1.6 Llama 3.1
Closest papers by meaning
- MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding (CVPR 2024 · similarity 0.91)
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.91)
- VTimeLLM: Empower LLM to Grasp Video Moments (CVPR 2024 · similarity 0.90)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.90)
- Understanding Long Videos with Multimodal Language Models (ICLR 2025 · similarity 0.90)
- Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.