Research map / 3D vision / Human pose estimation
Real-Time Simulated Avatar from Head-Mounted Sensors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Controls a physics-simulated humanoid avatar end-to-end from XR headset pose and monochrome SLAM camera images, trained by online distillation from a pretrained PHC motion imitator on a new synthetic Quest 2 dataset.
Problem
Estimating full-body global pose for an avatar from head-mounted sensors (headset 6DoF pose plus cameras) when the body is often out of view and headset pose alone lacks hand and foot detail.
Approach
Map images and headset pose directly to humanoid joint actuation end-to-end, avoiding kinematic intermediate representations, by distilling from a pretrained physics-based motion imitator (PHC) with DAgger-style supervised learning in simulation.
Key results
- On synthetic Quest 2 test, SimXR reaches 94.3% success and 66.4 Eg-mpjpe versus KinPoly-v's 82.2% and 66.7, with 59.7MB model size vs UnrealEgo's 554.5MB.
- On ADT (Aria) test, SimXR achieves 100% success and 47.7 Epa-mpjpe, vs KinPoly 60.8 and UnrealEgo 71.5.
- Training with distillation takes three days for 0.1B environment interactions, versus about 2B for PHC trained with RL over the same time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add vision input (ablation) | SimXR · Synthetic-Test | Succ: 73.0% -> 94.3% | Helped |
| add headset input (ablation) | SimXR · Synthetic-Test | Succ: 27.1% -> 94.3% | Helped |
| use group normalization instead of batch normalization (ablation) | SimXR · Synthetic-Test | Eg-mpjpe: 74.9 -> 66.4 | Helped |
| add distillation (ablation) | SimXR · Synthetic-Test | Eg-mpjpe: 226.1 -> 66.4 | Helped |
| add vision input (ablation) | SimXR · ADT-Test | Epa-mpjpe: 85.8 -> 47.7 | Helped |
| use SimXR instead of KinPoly-v | VR headset pose estimation · Synthetic-Test | Epa-mpjpe: 42.8 -> 40.0 | Helped |
| use SimXR instead of KinPoly-v | VR headset pose estimation · Real-world | Epa-mpjpe: 83.0 -> 73.0 | Helped |
| use SimXR instead of KinPoly | AR headset pose estimation · ADT-Test | Epa-mpjpe: 60.8 -> 47.7 | Helped |
Surprising findings
- Direct image-based RL without distillation achieves only 35.8% success on the synthetic test set, showing RL from pixels is impractical here.
Limitations
- Misplaced hands or feet when body parts are unobserved or blurry
- Humanoid lags behind real images when hands enter or leave view
- Single-frame design without temporal modeling
Experimental setup
Datasets: AMASS Aria Digital Twin (ADT) in-house MoCap (synthetic rendering) real-world Quest 2 capture (40k frames, pseudo-GT)
Benchmarks: Synthetic-Test (Quest 2) Real-world Quest 2 capture ADT-Test
Metrics: Succ Eg-mpjpe Empjpe Epa-mpjpe Eacc Evel
Baselines: UnrealEgo KinPoly-v KinPoly PHC
Builds on: PHC KinPoly UnrealEgo PULSE DAgger
Closest papers by meaning
- FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video (CVPR 2025 · similarity 0.87)
- HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations (CVPR 2024 · similarity 0.86)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.86)
- RAM-Avatar: Real-time Photo-Realistic Avatar from Monocular Videos with Full-body Control (CVPR 2024 · similarity 0.86)
- Stratified Avatar Generation from Sparse Observations (CVPR 2024 · similarity 0.86)
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.