Research map / 3D vision / Human pose estimation
EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EgoSim, a simulator for multiple body-worn cameras driven by real motion capture, and the MultiEgoView dataset (119 h synthetic plus 5 h real from six GoPro cameras), and trains a video-only multi-view transformer for ego-body pose estimation showing sim-to-real benefits.
Problem
Egocentric vision research has focused on head-mounted cameras, leaving body-worn cameras (wrists, knees, pelvis) under-explored, with few datasets and no simulators that model their motion artifacts and provide ground truth for full-body pose.
Approach
A configurable AirSim/Unreal-based simulator that attaches virtual cameras to any body part via spring-arm mounts driven by AMASS mocap, rendering synchronized RGB, depth, normals, segmentation and IMU data, paired with a real six-GoPro dataset with Xsens ground truth.
Key results
- Training on synthetic data only gives PA-MPJPE 0.040 m on synthetic test but 0.119 m on real data; training on real data alone gives global MPJPE 1.23 m.
- Synthetic pretraining followed by fine-tuning on real data reduces global MPJPE to 0.33 m (from 1.23 m) and PA-MPJPE to 0.044 m.
- A spring-damper camera mount yields 1.98 cm mean position error versus 2.35 cm for a rigid mount.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune synthetic-pretrained model on real data | Multi-view pose transformer · MultiEgoView, Real eval | Global MPJPE (m): 1.23 -> 0.40 | Helped |
| fine-tune synthetic-pretrained model on real data | Multi-view pose transformer · MultiEgoView, Real eval | PA-MPJPE (m): 0.087 -> 0.056 | Helped |
| fine-tune synthetic-pretrained model using 80% rather than 20% real data | Multi-view pose transformer · MultiEgoView, Real eval | Global MPJPE (m): 0.40 -> 0.33 | Helped |
| use all six body-worn cameras rather than fewer-camera setups (ablation) | Multi-view pose transformer · Synthetic scenes (1) and (2) | Global MPJPE: 0.238 -> 0.18; 0.293 -> 0.18; 0.345 -> 0.18 | Helped |
| use all six body-worn cameras rather than fewer-camera setups (ablation) | Multi-view pose transformer · Synthetic scenes (1) and (2) | PA-MPJPE: 0.051 -> 0.041; 0.06 -> 0.041; 0.0823 -> 0.041 | Helped |
| use a spring-damper camera mount instead of a rigid mount (ablation) | EgoSim camera mounting model · Loose body-worn GoPro tracking | Mean position error (cm): 2.35 cm -> 1.98 cm | Helped |
| train on one scene and evaluate on a different scene (ablation) | Multi-view pose transformer · Downtown (1) → CAB (2) | Global MPJPE: 0.18 -> 0.37 | Hurt |
| train on Synthetic (1) & (2) and evaluate on Synthetic (3) & (4) (ablation) | Multi-view pose transformer · Synthetic scene transfer | Global MPJPE: 0.18 -> 0.42 | Hurt |
| reduce real fine-tuning data from 80% to 20% | Synthetic-pretrained multi-view pose transformer · MultiEgoView, Real eval | MTE (m): 0.31 -> 0.37 | Hurt |
Surprising findings
- Synthetic-only training yields a low pose error on synthetic data but poor real-world global translation, and real-only training gives the worst global MPJPE.
- Fewer cameras produce lower jerk, indicating the model averages poses rather than capturing rapid motions.
Limitations
- Avatar animations are sampled independently from AMASS and do not model interactions with other humans or objects.
- Only four scenes are available, limiting generalization.
- Global root translation and orientation errors accumulate over long sequences due to relative prediction without localization grounding.
Experimental setup
Datasets: AMASS BABEL BEDLAM assets UCF101 MultiEgoView (synthetic + real)
Benchmarks: MultiEgoView
Metrics: Global MPJPE PA-MPJPE MTE MRE MJAE Jerk
Baselines: xR-EgoPose UnrealEgo EgoBody EgoHumans
Builds on: AirSim AMASS SMPL-X BABEL Sparse Video Tube ViT
Closest papers by meaning
- FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video (CVPR 2025 · similarity 0.91)
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.90)
- Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data (NeurIPS 2024 · similarity 0.90)
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation (NeurIPS 2025 · similarity 0.90)
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (CVPR 2024 · similarity 0.89)
- EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.