Research map / Video / Video-language understanding
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Ego-Exo4D, a large multimodal, multiview (ego + 4-5 exo) video dataset of skilled human activities with expert commentary and four families of benchmark tasks.
Problem
Existing ego-exo video datasets are small, unsynchronized, or staged, which limits research on understanding skilled human activity across first- and third-person viewpoints.
Approach
A time-synchronized, calibrated ego (Aria glasses) plus multiple exo (GoPro) capture rig recording unscripted skilled activities at many real-world sites, paired with multiple language corpora (expert commentary, narrate-and-act, atomic action descriptions) and benchmark tasks spanning relation, recognition, proficiency, and ego-pose.
Key results
- Dataset comprises 1,286 hours of ego and exo video, 5,035 takes, from 740 participants in 123 scenes across 13 cities.
- Expert commentary: 117,812 time-stamped, video-aligned pieces from 52 experts.
- Atomic action descriptions total 432K sentences; ego pose offers ~14M frames of 3D ground truth and pseudo-ground truth.
Limitations
- Focuses on single-person skilled activities only
- Data access gated behind a license system
- Baseline results are reported only in appendices, not in main text
Experimental setup
Datasets: Ego-Exo4D
Benchmarks: Ego-Exo4D benchmark tasks (correspondence, translation, keystep recognition, procedure understanding, proficiency estima
Baselines: CharadesEgo Assembly101 H2O Homage
Builds on: Ego4D HowTo100M Project Aria
Closest papers by meaning
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.89)
- EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World (CVPR 2024 · similarity 0.88)
- Retrieval-Augmented Egocentric Video Captioning (CVPR 2024 · similarity 0.88)
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation (NeurIPS 2025 · similarity 0.88)
- Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.87)
- Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.