Research map / 3D vision / Human pose estimation
Expressive Forecasting of 3D Whole-Body Human Motions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces whole-body 3D human motion forecasting (body plus both hands) and an Encoding-Alignment-Interaction (EAI) framework with cross-context alignment and cross-context interaction, evaluated on GRAB.
Problem
Existing human motion forecasting predicts only major body joints and ignores hand gestures, which carry fine-grained intent; jointly forecasting body and hands is hard due to heterogeneous motion scales and cross-part interactions.
Approach
Encodes body and hands separately (DCT plus GCN), aligns their heterogeneous latent distributions via a learnable cross-neutralization factor constrained by MMD (XCA), then models semantic and physical (wrist) interactions with a cross-attention variant (XCI).
Key results
- EAI achieves whole-body average MPJPE of 61.9 mm in the ablation full model, versus 68.7 mm without XCI.
- Under united training, EAI outperforms baselines (LTD, DMGNN, PGBIG, SPGSN) on major body, left/right hands at 0.2, 0.4 and 1.0 s; e.g., right-hand MPJPE at 1.0 s is 169.0 mm vs 195.6 mm for LTD (D).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EAI instead of LTD (U) | whole-body forecasting · GRAB, all actions, 0.2 sec | MPJPE (mm): 18.4 -> 16.7 | Helped |
| use EAI instead of LTD (U) | whole-body forecasting · GRAB, all actions, 0.4 sec | MPJPE (mm): 43.1 -> 40.7 | Helped |
| use EAI instead of LTD (U) | whole-body forecasting · GRAB, all actions, 1.0 sec | MPJPE (mm): 120.4 -> 104.6 | Helped |
| add cross neutralization (CN) (ablation) | EAI · GRAB, all actions, whole body | MPJPE (mm): 64.4 -> 61.9 | Helped |
| add discrepancy constraint (DC) (ablation) | EAI · GRAB, all actions, whole body | MPJPE (mm): 63.3 -> 61.9 | Helped |
| add cross neutralization and discrepancy constraint (CN and DC) (ablation) | EAI · GRAB, all actions, whole body | MPJPE (mm): 66.6 -> 61.9 | Helped |
| add physical interaction (PI) (ablation) | EAI · GRAB, all actions, whole body | MPJPE (mm): 64.5 -> 61.9 | Helped |
| add semantic interaction (SI) (ablation) | EAI · GRAB, all actions, whole body | MPJPE (mm): 67.5 -> 61.9 | Helped |
| add cross-context interaction (XCI) (ablation) | EAI · GRAB, all actions, whole body | MPJPE (mm): 68.7 -> 61.9 | Helped |
Surprising findings
- Training body and hands jointly as one whole-body graph hurts major-body prediction relative to divided training, indicating negative mutual interference without alignment.
Limitations
- Does not model interactions with objects, which the authors note could improve accuracy.
- No existing direct baselines for whole-body forecasting; baselines were adapted from body-only methods and retrained.
Experimental setup
Datasets: GRAB
Benchmarks: GRAB
Metrics: MPJPE MPJPE-AW
Baselines: LTD DMGNN PGBIG SPGSN
Builds on: LTD DMGNN PGBIG SPGSN SMPL-X
Closest papers by meaning
- Forecasting of 3D Whole-body Human Poses with Grasping Objects (CVPR 2024 · similarity 0.94)
- MotionMap: Representing Multimodality in Human Pose Forecasting (CVPR 2025 · similarity 0.88)
- SimpliHuMoN: Simplifying Human Motion Prediction (NeurIPS 2026 · similarity 0.88)
- HumanTOMATO: Text-aligned Whole-body Motion Generation (ICML 2024 · similarity 0.88)
- FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observations (CVPR 2024 · similarity 0.87)
- Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.