Research map / 3D vision / Human pose estimation
EnvPoser: Environment-aware Realistic Human Motion Estimation from Sparse Observations with Uncertainty Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EnvPoser, a two-stage model that estimates full-body motion from HMD and controller signals by sampling uncertainty-aware multi-hypothesis initial estimates and refining them with semantic and geometric constraints from a pre-scanned environment point cloud.
Problem
Full-body motion estimation from sparse VR tracking signals (head and hands) is ill-posed, with many plausible lower-body motions per input; existing methods mostly ignore rich human-environment interaction cues.
Approach
Model multi-hypothesis motion via heteroscedastic joint uncertainty with reparameterized sampling in stage I (trained on AMASS), then refine the hypotheses in stage II using cross-attention to a PointNet++-encoded scene for semantic cues plus COAP-based collision and foot-ground losses for geometric constraints.
Key results
- On EgoBody, EnvPoser reaches 6.00 MPJRE, 74.7 mm MPJPE and 6.6 Jitter, beating AvatarJLM (6.42, 91.7, 7.2).
- On GIMO, EnvPoser reaches 4.38 MPJRE and 57.6 mm MPJPE, versus S2Fusion's 4.65 and 57.8.
- Ablation on EgoBody: removing uncertainty estimation raises MPJPE from 74.7 to 80.1 mm in the full refinement pipeline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add environment-aware refinement using semantic and geometric constraints (ablation) | uncertainty-aware motion estimation · EgoBody | MPJPE: 87.1 -> 74.7 | Helped |
| add environment-aware refinement using semantic and geometric constraints (ablation) | uncertainty-aware motion estimation · GIMO | MPJPE: 74.6 -> 57.6 | Helped |
| add environment-semantic refinement (ablation) | EnvPoser · EgoBody | MPJPE: 78.0 -> 74.7 | Helped |
| add environment-semantic refinement (ablation) | EnvPoser · GIMO | MPJPE: 62.5 -> 57.6 | Helped |
| add environment-geometric refinement (ablation) | EnvPoser · EgoBody | MPJPE: 77.5 -> 74.7 | Helped |
| add environment-geometric refinement (ablation) | EnvPoser · GIMO | MPJPE: 63.2 -> 57.6 | Helped |
| add joint uncertainty estimation (ablation) | baseline motion estimator · EgoBody | MPJPE: 90.8 -> 87.1 | Helped |
| add joint uncertainty estimation (ablation) | baseline motion estimator · GIMO | MPJPE: 77.5 -> 74.6 | Helped |
| add joint uncertainty estimation and resampling (ablation) | EnvPoser · EgoBody | MPJRE: 6.56 -> 6.00 | Helped |
| add joint uncertainty estimation and resampling (ablation) | EnvPoser · GIMO | MPJPE: 60.2 -> 57.6 | Helped |
| use EnvPoser instead of AvatarJLM | motion estimation · EgoBody | MPJPE: 91.7 -> 74.7 | Helped |
| use EnvPoser instead of S2Fusion | motion estimation · GIMO | MPJRE: 4.65 -> 4.38 | Helped |
Limitations
- Assumes a static environment; no handling of other people or moving objects.
- Real-time mesh quality in complex environments is limited and can affect performance.
- S2Fusion was not open-source and had to be reproduced by the authors.
Experimental setup
Datasets: AMASS EgoBody GIMO
Benchmarks: EgoBody GIMO
Metrics: MPJRE MPJPE MPJVE Jitter
Baselines: AvatarPoser AGRoL AvatarJLM S2Fusion
Builds on: AvatarPoser AvatarJLM S2Fusion EgoHMR COAP
Closest papers by meaning
- A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signals (CVPR 2024 · similarity 0.87)
- From Sparse Signal to Smooth Motion: Real-Time Motion Generation with Rolling Prediction Models (CVPR 2025 · similarity 0.86)
- DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion (CVPR 2024 · similarity 0.86)
- HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations (CVPR 2024 · similarity 0.85)
- Dynamic Inertial Poser (DynaIP): Part-Based Motion Dynamics Learning for Enhanced Human Pose Estimation with Sparse Inertial Sensors (CVPR 2024 · similarity 0.85)
- UMotion: Uncertainty-driven Human Motion Estimation from Inertial and Ultra-wideband Units (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.