Research map / Video / Action recognition
EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EgoExoLearn, a 120-hour dataset of egocentric videos recorded while participants replicate tasks from exocentric demonstration videos, with gaze and multi-level annotations, and four cross-view benchmarks.
Problem
Existing ego-exo datasets record synchronized, same-environment views, so there is no resource for bridging asynchronous procedural activities where a person follows a demonstration recorded in a different place and viewpoint.
Approach
A demonstration-following setting in which egocentric videos are recorded while participants replicate exocentric tutorial videos in a different environment, with calibrated eye gaze and fine-grained verb/noun/hand annotations, plus benchmarks for cross-view association, action anticipation/planning, and referenced skill assessment.
Key results
- In cross-view association, the EgoExo model with gaze reaches 55.3% Ego2Exo and 51.1% Exo2Ego on test, versus 30.6% for best zero-shot model (EgoVLP 32.1%/28.9%, random ~14%).
- Co-training with ego and exo data achieves the best cross-view action anticipation/planning results in all setups, and gaze consistently reduces the ego-exo gap (e.g., Ego-only anticipation Ego-V rising from 33.4 to 40.5 with gaze).
- Referenced skill assessment with RAAN plus triplet loss and gaze reaches about 82-84% ranking accuracy across tasks, with the exo reference giving only marginal gains over ego-only pairs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add gaze-cropped ego video during zero-shot InternVideo transfer (ablation) | InternVideo · Cross-view association test, Ego2Exo | Top-1 accuracy: 27.0 -> 30.9 | Helped |
| Add gaze-cropped ego video during zero-shot InternVideo transfer (ablation) | InternVideo · Cross-view association test, Exo2Ego | Top-1 accuracy: 21.7 -> 32.2 | Helped |
| Add gaze when co-training ego and exo data (ablation) | EgoExo · Cross-view association test, Ego2Exo | Top-1 accuracy: 49.0 -> 55.3 | Helped |
| Add gaze when co-training ego and exo data (ablation) | EgoExo · Cross-view association test, Exo2Ego | Top-1 accuracy: 45.3 -> 51.1 | Helped |
| Add gaze during co-training (ablation) | Ego & Exo · Cross-view anticipation test, Ego-V | Class-mean Top-5 recall: 33.5 -> 43.8 | Helped |
| Add gaze during co-training (ablation) | Ego & Exo · Cross-view anticipation test, Ego-N | Class-mean Top-5 recall: 37.4 -> 53.3 | Helped |
| Add gaze during co-training (ablation) | Ego & Exo · Cross-view anticipation test, Exo-V | Class-mean Top-5 recall: 39.6 -> 40.3 | Helped |
| Add gaze during co-training (ablation) | Ego & Exo · Cross-view anticipation test, Exo-N | Class-mean Top-5 recall: 44.3 -> 44.4 | Helped |
| Add gaze during co-training (ablation) | Ego & Exo · Cross-view planning test, Ego | ED@K: 83.2 -> 79.0 | Helped |
| Add gaze during co-training (ablation) | Ego & Exo · Cross-view planning test, Exo | ED@K: 76.0 -> 75.6 | Helped |
| Add gaze to ego-pair skill assessment (ablation) | RAAN · Cross-view skill assessment, Egg Cracking | Ranking accuracy: 78.45 -> 82.08 | Helped |
| Add gaze to ego-pair skill assessment (ablation) | RAAN · Cross-view skill assessment, Peeling | Ranking accuracy: 78.52 -> 79.37 | Helped |
Surprising findings
- Adding the exocentric reference video yields only marginal improvement in skill assessment, suggesting models struggle to bridge asynchronous views.
- Zero-shot video-language models perform poorly on cross-view association, and some (e.g., LaViLa, InternVideo with gaze) show mixed effects from gaze cropping.
Limitations
- Cross-view association training set has no paired ego-exo samples, so the benchmark relies on unpaired training data.
- Demonstration videos are sourced from online platforms and lab tutorials, limiting diversity of scenarios to 5 daily and 3 lab tasks.
- Gaze benefit is described as naively designed; models still struggle to fully bridge views.
Experimental setup
Datasets: EgoExoLearn Ego4D EgoVLP pre-training data
Benchmarks: EgoExoLearn cross-view association EgoExoLearn cross-view action anticipation EgoExoLearn cross-view action planning EgoExoLearn cross-view referenced skill assessment
Metrics: Top-1 accuracy class-mean Top-5 recall edit distance ED@K
Baselines: EgoVLP LaViLa InternVideo CLIP + TA3N RAAN Who's better
Builds on: EgoVLP CLIP TA3N RAAN LaViLa
Closest papers by meaning
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation (NeurIPS 2025 · similarity 0.90)
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (CVPR 2024 · similarity 0.88)
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.88)
- EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.88)
- Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos (CVPR 2026 · similarity 0.86)
- EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.