Research map / 3D vision / Human pose estimation
FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Jointly forecasts sequences of coarse action labels and characteristic 3D human poses from 2D RGB videos, using a differentiable 2D projection loss and an adversarial 3D prior trained on unpaired 3D pose data.
Problem
Forecasting long-term complex human behavior (sequences of sub-actions) in 3D when no paired 3D pose and action annotations exist; only 2D action videos are readily available.
Approach
An autoregressive generator jointly predicts the next action label and a characteristic 3D pose, supervised only by 2D reprojection against future video poses and regularized by a Wasserstein adversarial critic trained on uncorrelated 3D skeleton databases.
Key results
- On MPII Cooking II, the joint method reaches 50 px 2D MPJPE and 29% top-1 / 51% top-3 action accuracy, versus 55 px and 27%/43% for joint 2D action-pose forecasting [95].
- On IKEA-ASM, the method achieves 40 px MPJPE and 29% top-1 action accuracy, outperforming AVT and FUTR action-only baselines (22-23% top-1).
- Ablation shows characteristic poses outperform uncoupled, middle, and random pose types on MPII Cooking II (50 px vs 75 px MPJPE for uncoupled).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly predict action labels and characteristic 3D poses rather than forecasting them separately | FutureHuman3D · MPII Cooking II | MPJPE [px]: 55 -> 50 | Helped |
| Jointly predict action labels and characteristic 3D poses rather than forecasting them separately | FutureHuman3D · IKEA-ASM | MPJPE [px]: 44 -> 40 | Helped |
| Use characteristic poses rather than poses sampled independently of actions (ablation) | FutureHuman3D · MPII Cooking II | MPJPE [px]: 75 -> 50 | Helped |
| Use characteristic poses rather than poses sampled independently of actions (ablation) | FutureHuman3D · MPII Cooking II | top-1 action accuracy: 28% -> 29% | Helped |
| Use poses from the middle of the action range rather than characteristic poses (ablation) | FutureHuman3D · MPII Cooking II | MPJPE [px]: 50 -> 58 | Hurt |
| Use randomly selected poses within the action range rather than characteristic poses (ablation) | FutureHuman3D · MPII Cooking II | top-1 action accuracy: 29% -> 22% | Hurt |
| Add 2D projection and 3D adversarial losses to action-only training (ablation) | FutureHuman3D · MPII Cooking II | top-1 action accuracy: 21% -> 29% | Helped |
| Add 2D projection supervision to action-only training (ablation) | FutureHuman3D · MPII Cooking II | top-1 action accuracy: 21% -> 26% | Helped |
| Add 3D adversarial regularization to 2D projection training (ablation) | FutureHuman3D · MPII Cooking II | 3D pose quality: 0.10 -> 0.53 | Helped |
| Add 3D adversarial regularization to action and 2D projection training (ablation) | FutureHuman3D · IKEA-ASM | MPJPE [px]: 46 -> 40 | Helped |
| Use joint action and 3D pose forecasting rather than action forecasting alone (ablation) | FutureHuman3D · MPII Cooking II | top-1 action accuracy: +8 percentage points | Helped |
Surprising findings
- Forecasting poses jointly with actions improves action accuracy over action-only training, suggesting pose forecasting aids long-term action understanding.
Limitations
- Relies on a separate 2D pose extraction step rather than end-to-end learning from video.
- Uses only 9 upper-body joints, so hands and fine-grained contact are not modeled.
- 3D quality is only measured relative to a pose database, with no 3D ground truth for action sequences.
Experimental setup
Datasets: MPII Cooking II IKEA-ASM Human3.6M AMASS GRAB
Benchmarks: MPII Cooking II IKEA-ASM
Metrics: 2D MPJPE 3D pose quality (classifier-based) top-1 action accuracy top-3 action accuracy
Baselines: DLow GSPS STARS EqMotion AVT FUTR Zhu et al. joint 2D action and pose forecasting Zero Velocity Train Average
Builds on: Diller et al. characteristic 3D pose forecasting WGAN RepNet weakly supervised 3D pose estimation OpenPose
Closest papers by meaning
- Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning (CVPR 2024 · similarity 0.88)
- VideoMDM: Towards 3D Human Motion Generation From 2D Supervision (NeurIPS 2026 · similarity 0.88)
- Harmonizing Stochasticity and Determinism: Scene-responsive Diverse Human Motion Prediction (NeurIPS 2024 · similarity 0.87)
- Expressive Forecasting of 3D Whole-Body Human Motions (AAAI 2024 · similarity 0.87)
- MotionMap: Representing Multimodality in Human Pose Forecasting (CVPR 2025 · similarity 0.87)
- Lifting Motion to the 3D World via 2D Diffusion (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.