Research map / 3D vision / 3D generation
EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EgoExOR, a synchronized ego-exo operating-room dataset of 84,553 frames with scene graph annotations, plus a dual-branch LLM-based scene graph generation baseline.
Problem
Existing operating room datasets provide either partial egocentric views or sparse exocentric multi-view context, lacking a combination of egocentric and exocentric perspectives with gaze, hand pose, and dense scene graphs for surgical scene understanding.
Approach
Combine multi-view egocentric recordings from wearable glasses (RGB, gaze, hand tracking, audio) with exocentric RGB-D cameras and ultrasound screen capture in a simulated spine-surgery OR, and adapt a dual-branch LLM-based scene graph generator that fuses ego and exo streams separately.
Key results
- EgoExOR model achieves overall macro F1 0.72 vs 0.67 for MM2SG and 0.63 for ORacle on scene graph generation.
- Exocentric-only ablation drops overall macro F1 to 0.42, while egocentric-only reaches 0.68 versus 0.72 for the full model.
- Dataset comprises 41 takes, 94 minutes, 84,553 frames at 15 FPS, and 568,235 relation triplets with 36 entity and 22 relation classes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add gaze and hand pose and use a dual-branch ego-exo architecture alongside MM2SG's modalities | MM2SG · UI | macro F1: 0.72 -> 0.79 | Helped |
| Add gaze and hand pose and use a dual-branch ego-exo architecture alongside MM2SG's modalities | MM2SG · MISS | macro F1: 0.66 -> 0.68 | Helped |
| Add gaze and hand pose and use a dual-branch ego-exo architecture alongside MM2SG's modalities | MM2SG · Overall | macro F1: 0.67 -> 0.72 | Helped |
| Add exocentric RGB and point clouds to egocentric inputs (ablation) | EgoExOR · UI | macro F1: 0.71 -> 0.79 | Helped |
| Add exocentric RGB and point clouds to egocentric inputs (ablation) | EgoExOR · MISS | macro F1: 0.67 -> 0.68 | Helped |
| Add exocentric RGB and point clouds to egocentric inputs (ablation) | EgoExOR · Overall | macro F1: 0.68 -> 0.72 | Helped |
| Add egocentric RGB, gaze, and hand pose to exocentric inputs (ablation) | EgoExOR · UI | macro F1: 0.45 -> 0.79 | Helped |
| Add egocentric RGB, gaze, and hand pose to exocentric inputs (ablation) | EgoExOR · MISS | macro F1: 0.41 -> 0.68 | Helped |
| Add egocentric RGB, gaze, and hand pose to exocentric inputs (ablation) | EgoExOR · Overall | macro F1: 0.42 -> 0.72 | Helped |
| Use egocentric inputs instead of exocentric inputs (ablation) | EgoExOR · Overall | macro F1: 0.42 -> 0.68 | Helped |
Surprising findings
- Exocentric-only input performs far worse (0.42) than egocentric-only (0.68), with the authors noting the gap in predicates like inserting and injecting.
- Low-frequency predicates such as cutting and anaesthetising remain hard, with near-zero F1 for anaesthetising (0.02).
Limitations
- Simulated procedures with biomedical engineers as actors may not reflect expert surgical performance in live clinical settings.
- Recorded in a single specially equipped OR, which may limit adaptability to other OR configurations.
- Low-frequency predicates such as cutting and anaesthetising are poorly recognized.
Experimental setup
Datasets: EgoExOR
Benchmarks: EgoExOR
Metrics: macro F1 (predicates)
Baselines: ORacle MM2SG
Builds on: ORacle MM2SG LLaVA LoRA
Closest papers by meaning
- MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments (CVPR 2025 · similarity 0.88)
- CholecTrack20: A Multi-Perspective Tracking Dataset for Surgical Tools (CVPR 2025 · similarity 0.86)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.85)
- EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World (CVPR 2024 · similarity 0.85)
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (CVPR 2024 · similarity 0.84)
- Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.