Research map / Video / Video-language understanding
Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoMindPalace, a three-layer topological semantic graph (human-object interactions, activity zones, room layout) built from perception models and serialized as JSON for text-only LLMs, plus the VMB benchmark for long egocentric video QA.
Problem
Long-form video understanding with LLMs is limited by bounded context windows and information overload when key moments are temporally dispersed but spatially concentrated.
Approach
Organize video into a hierarchical graph whose nodes are objects/humans, activity zones discovered via CLIP features and camera pose, and rooms, with spatial, interaction, and temporal edges, so that a text-only LLM can reason over a compact JSON representation grounded in 3D layout.
Key results
- Achieves 75.8% on NExT-QA, 70.1% on EgoSchema (text says 68.6%), and 70.1% on IntentQA, outperforming VideoTree on the standard benchmarks.
- On VMB, averages 51.2 Spatial, 56.3 Temporal, 34.5 Layout, and 3.8 open-ended score, the best among compared methods.
- Reaches 37.8% total accuracy on Active Memories Benchmark, above VideoAgent (34.6%) and AMEGO (36.3%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a three-layer semantic graph for video representation instead of VideoTree's tree representation | VideoMindPalace on NExT-QA · NExT-QA | accuracy: 73.5 -> 75.8 | Helped |
| Use a three-layer semantic graph for video representation instead of VideoTree's tree representation | VideoMindPalace on EgoSchema · EgoSchema | accuracy: 66.2 -> 70.1 | Helped |
| Use a three-layer semantic graph for video representation instead of VideoTree's tree representation | VideoMindPalace on IntentQA · IntentQA | accuracy: 66.9 -> 70.1 | Helped |
| Represent long video as a layered topological semantic graph | VideoMindPalace on VMB · VMB Spatial (Avg) | accuracy: 47.0 -> 51.2 | Helped |
| Represent long video as a layered topological semantic graph | VideoMindPalace on VMB · VMB Temporal (Avg) | accuracy: 54.9 -> 56.3 | Helped |
| Represent long video as a layered topological semantic graph | VideoMindPalace on VMB · VMB Layout (Avg) | accuracy: 24.5 -> 34.5 | Helped |
| Represent long video as a layered topological semantic graph | VideoMindPalace on VMB · VMB Open-Ended | score (0-5): 3.3 -> 3.8 | Helped |
| Use VideoMindPalace's graph-based representation | VideoMindPalace on Active Memories Benchmark · Active Memories Benchmark | accuracy (Total): 36.3 -> 37.8 | Helped |
Surprising findings
- Lags behind AMEGO on certain spatial grounding AMB tasks due to hand-side prediction errors when converting visual queries to text.
Limitations
- Errors from perception models (tracking, captioning) propagate into the graph.
- Does not encode low-level visual attributes such as color or texture, so color-based queries can fail.
- Relies on text-only LLM with captions, losing visual detail for visual-crop queries.
Experimental setup
Datasets: EgoSchema NExT-QA IntentQA Active Memories Benchmark EPIC-KITCHENS Ego-4D
Benchmarks: NExT-QA EgoSchema IntentQA Active Memories Benchmark VMB
Metrics: accuracy LLM-assisted open-ended score (0-5)
Baselines: IG-VLM LLoVi VideoTree VideoAgent Video-LLaMA2 SF-LLAVA-34B AMEGO S-QA
Builds on: VideoAgent VideoTree AMEGO Ego-TOPO Mind Palace technique
Closest papers by meaning
- Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition (ICML 2024 · similarity 0.89)
- SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge (CVPR 2024 · similarity 0.88)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.88)
- Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding (NeurIPS 2025 · similarity 0.88)
- STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training (CVPR 2025 · similarity 0.88)
- TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.