Research map / 3D vision / 3D generation
EgoGen: An Egocentric Synthetic Data Generator
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents EgoGen, a scalable synthetic data generator for egocentric (HMD) perception that uses a closed-loop, egocentric-sensing-driven human motion model trained with two-stage PPO over collision-avoiding motion primitives, and shows gains on mapping/localization, camera tracking, and human mesh recovery.
Problem
Egocentric perception for AR/HMD suffers from scarce labeled data, and synthetic data generation lacks realistic, embodied human movements that steer the egocentric camera through 3D scenes.
Approach
Virtual humans perceive the environment through a cheap egocentric sensing proxy (ray casts from the eye position) and a policy selects composable collision-avoiding motion primitives in a learned latent action space, with an attention reward for looking toward targets, removing the need for pre-computed global paths.
Key results
- In moving-obstacle and 4-human scenes, EgoGen's CAMPs reach 100% success with 0.06-0.07 m final distance, versus 83-96% and 0.29-0.79 m for GAMMA and DIMOS baselines.
- Pretraining with EgoGen data raises LaMAR localization recall from 66.9% to 76.7%.
- Depth-based HMR on EgoBody with EgoGen pretraining lowers G-MPJPE by 22.9% (117.7 to 90.7 mm) relative to training from scratch.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EgoGen motion synthesis instead of GAMMA* | Motion synthesis · Moving obstacles | SR (%): 96 -> 100 | Helped |
| use EgoGen motion synthesis instead of DIMOS* | Motion synthesis · Moving obstacles | Dist. (m): 0.55 -> 0.06 | Helped |
| use EgoGen motion synthesis instead of GAMMA* | Motion synthesis · 2 humans | SR (%): 95 -> 100 | Helped |
| use EgoGen motion synthesis instead of DIMOS* | Motion synthesis · 4 humans | Dist. (m): 0.79 -> 0.07 | Helped |
| use compact egocentric sensing instead of a local map | CAMP policy · Replica | SR (%): 78 -> 95 | Helped |
| add EgoGen images to LaMAR mapping data | LaMAR mapping and localization · LaMAR CAB validation | Recall (%): 66.9 -> 76.7 | Helped |
| add EgoGen synthetic pretraining | Egocentric camera tracking · GIMO | Pose: 1.83 -> 1.67 | Helped |
| add EgoGen synthetic pretraining | Depth-based ProHMR · EgoBody test set, depth | G-MPJPE (mm): 117.7 -> 90.7 | Helped |
| add EgoGen synthetic pretraining | RGB-based ProHMR · EgoBody test set, RGB | MPJPE (mm): 90.7 -> 85.3 | Helped |
| use compact egocentric sensing instead of egocentric depth images (ablation) | CAMP policy · Replica | SR (%): 8 -> 92 | Helped |
| add policy pretraining before strict penetration-constraint training (ablation) | CAMP policy · Replica | ∥VP∥2: 28.77 -> 10.57 | Helped |
| add the attention reward (ablation) | CAMP policy · Replica | cos(#»v, #»a): 0.891 -> 0.940 | Helped |
Surprising findings
- Pretraining with EgoGen synthetic data improves localization recall but yields no gain in translation error for egocentric camera tracking (1303 vs 1305 mm).
Limitations
- Human-scene interaction is coarse (no hand manipulation, sitting, or lying).
- Attention goals are fixed rather than predicted from human intent.
- Synthetic images are noisier due to scene quality, per the mapping experiments.
Experimental setup
Datasets: SAMP AMASS BEDLAM Replica LaMAR EgoBody GIMO
Benchmarks: LaMAR GIMO EgoBody Replica
Metrics: success rate distance to goal contact score penetration percentage localization recall at (1 deg, 10 cm) track length pose error MPJPE G-MPJPE PA-MPJPE
Baselines: GAMMA DIMOS Ng et al. data augmentation ProHMR EgoEgo LaMAR
Builds on: GAMMA DIMOS C-VAE motion primitives Proximal Policy Optimization BEDLAM
Closest papers by meaning
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation (NeurIPS 2025 · similarity 0.91)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.90)
- Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span (NeurIPS 2025 · similarity 0.89)
- PlayerOne: Egocentric World Simulator (NeurIPS 2025 · similarity 0.89)
- Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation (NeurIPS 2026 · similarity 0.88)
- FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.