Research map / 3D vision / 3D generation
A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S2Fusion, a conditional diffusion model that estimates scene-aware full-body motion from sparse HMD and hand-controller signals, using a motion-prior initialization, periodic-autoencoder features, and scene-penetration and phase-matching loss guidance.
Problem
Estimating plausible full-body human motion from sparse 6D tracking signals (head and hands) in 3D scenes is highly ambiguous, and lower-body motion is unconstrained because no lower-body observations exist, leading to scene penetration and leg-hand incoordination.
Approach
Combine scene geometry with sparse tracking signals in a conditional diffusion model initialized from a VAE motion prior, add periodic-autoencoder time-alignment features, and guide sampling with a scene-penetration loss and a phase-matching loss to regularize the legs.
Key results
- On CIRCLE, S2Fusion reaches MPJPE 19.2 mm vs 24.6 mm for AvatarJLM (the best baseline), and on GIMO MPJPE 57.8 mm vs 70.7 mm.
- Lower-body position error on CIRCLE drops to 57.3 mm from 69.1 mm for AvatarJLM.
- Ablations show scene input, motion prior, and PAE each improve MPJPE; loss guidance gives a modest further gain (CIRCLE MPJPE 20.6 to 19.2 with both losses).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add scene information as an input (ablation) | S2Fusion without motion prior or PAE · CIRCLE | MPJPE: 26.2 -> 22.7 | Helped |
| add scene information as an input (ablation) | S2Fusion without motion prior or PAE · GIMO | MPJPE: 76.6 -> 68.1 | Helped |
| add a pre-trained motion prior (ablation) | S2Fusion with scene input and without PAE · CIRCLE | MPJPE: 22.7 -> 20.9 | Helped |
| add a pre-trained motion prior (ablation) | S2Fusion with scene input and without PAE · GIMO | MPJPE: 68.1 -> 60.3 | Helped |
| add the periodic autoencoder (ablation) | S2Fusion with scene input and without motion prior · CIRCLE | MPJPE: 22.7 -> 21.8 | Helped |
| add the periodic autoencoder (ablation) | S2Fusion with scene input and without motion prior · GIMO | MPJPE: 68.1 -> 65.9 | Helped |
| add the scene-penetration loss (ablation) | S2Fusion without loss guidance · CIRCLE | MPJPE: 20.6 -> 20.1 | Helped |
| add the scene-penetration loss (ablation) | S2Fusion without loss guidance · GIMO | MPJPE: 59.9 -> 58.3 | Helped |
| add the phase-matching loss (ablation) | S2Fusion without loss guidance · CIRCLE | MPJPE: 20.6 -> 19.8 | Helped |
| add the phase-matching loss (ablation) | S2Fusion without loss guidance · GIMO | MPJPE: 59.9 -> 57.6 | Helped |
Surprising findings
- Phase-matching and scene-penetration losses can increase jitter (e.g., Jitter rises to 6.1 with phase loss alone on CIRCLE vs 5.6 baseline).
Limitations
- Evaluated only on two motion-scene datasets (CIRCLE and GIMO) with a limited scene set.
- Loss guidance can introduce jittering motions.
- Physically plausible constraints for more complex scenarios are left to future work.
Experimental setup
Datasets: CIRCLE GIMO AMASS
Benchmarks: CIRCLE GIMO
Metrics: MPJRE MPJPE MPJVE Jitter FS Hand PE Upper PE Lower PE
Baselines: AvatarPoser AGRoL AvatarJLM
Builds on: DDPM-style motion diffusion (MDM) periodic autoencoder loss-guided diffusion AMASS
Closest papers by meaning
- EnvPoser: Environment-aware Realistic Human Motion Estimation from Sparse Observations with Uncertainty Modeling (CVPR 2025 · similarity 0.87)
- From Sparse Signal to Smooth Motion: Real-Time Motion Generation with Rolling Prediction Models (CVPR 2025 · similarity 0.87)
- Seamless Human Motion Composition with Blended Positional Encodings (CVPR 2024 · similarity 0.86)
- Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data (NeurIPS 2024 · similarity 0.86)
- ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos (CVPR 2025 · similarity 0.85)
- Full-Body Motion Reconstruction with Sparse Sensing from Graph Perspective (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.