Research map / Video / Video-language understanding

Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs

NeurIPS 2026 · Video › Video-language understanding · approach: Event and anomaly reasoning in video · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SG-Ego, a 3.8M spatio-temporal scene graph annotation set over Ego4D, and GLEN, a graph-language model that aligns scene graphs with activity text and forecasts activity-conditioned graph edits (A-GEF).

Problem

Egocentric video understanding relies on implicit embeddings, which makes it hard to explicitly reason about how human activities change the scene over time.

Approach

Represent egocentric scenes as consolidated spatio-temporal scene graphs and model their evolution as activity-conditioned graph edits, extending Graph Edit Networks with a text-conditioned edit model and node query slots for future objects.

Key results

• GLEN achieves 15.3 global R@1 and 47.7 local R@1 on EgoCVR, outperforming TFR-CVR (14.1 / 44.2) without CVR training.
• On A-GEF, GLEN reaches 48.49 triplet R@100 versus 23.17 for the static graph baseline and 9.14 for Qwen3.5-9B.
• On EXPLORE-Bench Full, GLEN obtains S_obj 65.59, above Qwen3-VL-8B-Thinking (62.70), though its S_rel (2.69) trails Qwen3-VL-8B-Instruct (2.82).

Surprising findings

• Qwen3.5-9B given SG-Ego graphs (66.0%) nearly matches its performance on raw frames (72.8%) on EgoSchema.
• The Qwen3.5-9B LLM baseline on A-GEF predicts few triplets, leaving recall constant across K.

Limitations

• Scene graph annotations are produced by an MLLM and open-vocabulary grounding pipeline, so label noise is likely and unquantified.
• Vocabulary is closed (1480 objects, 387 relations), and the A-GEF setting omits explicit node replacement.
• GLEN underperforms on intra-video EgoMCQ and on S_rel in EXPLORE-Bench relative to some baselines.

Experimental setup

Datasets: Ego4D EgoClip EgoSchema EgoMCQ EgoCVR EXPLORE-Bench PVSG

Benchmarks: EgoSchema EgoMCQ EgoCVR A-GEF (SG-Ego Edit) EXPLORE-Bench

Metrics: accuracy Inter/Intra-video accuracy Recall@K triplet Recall@K S_obj S_rel

Baselines: EgoVLP HierVL EgoVLPv2 HiERO TFR-CVR CoVR CIReVL Qwen3.5-9B GPT-5.2-Chat Gemini-3-Pro Qwen3-VL-8B-Thinking

Builds on: Graph Edit Networks TripletGCN EgoVLP Ego4D GroundingDINO

Closest papers by meaning

• Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.90)
• ESCA: Contextualizing Embodied Agents via Scene-Graph Generation (NeurIPS 2025 · similarity 0.89)
• SAW-Bench: Learning Situated Awareness in the Real World (ICML 2026 · similarity 0.88)
• EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT (NeurIPS 2025 · similarity 0.87)
• PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning (ICML 2025 · similarity 0.87)
• DynaMind: Reasoning over Abstract Video Dynamics for Embodied Decision-Making (ICML 2025 · similarity 0.87)