Research map / Video / Action recognition
Action Scene Graphs for Long-Form Understanding of Egocentric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Egocentric Action Scene Graphs (EASGs), a temporally evolving graph representation of camera-wearer actions, releases the Ego4D-EASG annotations, and shows a simple baseline plus LLM-based anticipation and summarization gains over verb-noun representations.
Problem
Existing egocentric video annotations (verb-noun labels, short-range boxes and states) lack a structured, long-form representation of how actions, interacted objects and their relationships unfold over time.
Approach
Represent egocentric actions as time-varying directed graphs whose nodes are the camera wearer, the action verb and grounded objects, with edges encoding direct-object, action and preposition relations, and extend Ego4D with manually annotated graphs collected via a multi-annotator pipeline with validation.
Key results
- Baseline EASG Cls achieves R@20 of 18.3 (no constraint) and 16.4 (with constraint), well above random guess (0.5 and 0.0).
- In action anticipation with GPT-3 text-davinci-003, EASG at T=20 reaches Top-5 verb 15.97 vs 8.41 for verb-noun, and Top-5 action 9.24 vs 4.98.
- In activity summarization with GPT-3.5 Turbo, EASG improves CIDEr from 9.42 (verb-noun) to 13.79, below narrations at 19.99.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EASG rather than verb-noun sequences for T=20 action anticipation | GPT3 text-davinci-003 · Action anticipation, T=20 | Verb Top-5 accuracy: +7.56 | Helped |
| Use EASG rather than verb-noun sequences for T=20 action anticipation | GPT3 text-davinci-003 · Action anticipation, T=20 | Noun Top-5 accuracy: +2.41 | Helped |
| Use EASG rather than verb-noun sequences for T=20 action anticipation | GPT3 text-davinci-003 · Action anticipation, T=20 | Action Top-5 accuracy: +4.26 | Helped |
| Use EASG rather than verb-noun sequences for activity summarization | GPT-3.5 Turbo · Activity summarization | CIDEr | Helped |
Surprising findings
- EASG Cls scores are much lower than edge and scene graph classification, showing verb prediction adds substantial difficulty.
Limitations
- Baseline is a simple fully-connected model on pre-extracted features, so generation performance is low.
- Annotation covers only 221 clips, and only 1 of 3 annotations was validated for disagreements in some cases.
- Downstream experiments rely on a small subset (3030 samples and 147 clips) and on closed LLM APIs.
Experimental setup
Datasets: Ego4D Ego4D-EASG
Benchmarks: Ego4D-EASG
Metrics: Recall@K top-k accuracy CIDEr ROUGE-1 ROUGE-2 ROUGE-L BLEU-1 BLEU-2 BLEU-3 BLEU-4
Baselines: Random Guess Verb-noun sequence representation
Builds on: Action Genome Ego4D SlowFast Faster R-CNN
Closest papers by meaning
- Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs (NeurIPS 2026 · similarity 0.90)
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation (NeurIPS 2025 · similarity 0.89)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.87)
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (CVPR 2024 · similarity 0.87)
- Retrieval-Augmented Egocentric Video Captioning (CVPR 2024 · similarity 0.87)
- The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.