Research map / 3D vision / Human pose estimation
Multimodal Sense-Informed Forecasting of 3D Human Motions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SIF3D, a multimodal sense-informed predictor that forecasts future 3D human motion conditioned on a scene point cloud and human gaze, using ternary intention-aware and semantic coherence-aware attention.
Problem
Forecasting future 3D human poses and trajectories in real 3D scenes while producing physically plausible motions that respect the surrounding environment and the person's intention.
Approach
Encodes the scene point cloud per point and uses two cross-modal attentions: TIA fuses motion, gaze and global scene salience for trajectory planning, while SCA scores local per-frame scene point salience with a spatial bias for pose prediction; a geometry discriminator enforces physical plausibility.
Key results
- On GIMO, SIF3D reaches 590 mm Traj-path and 666 mm Traj-dest with scene+gaze, versus 661/727 for BiFu.
- On GTA-1M, SIF3D achieves 836 mm Traj-dest and 227.7 mm MPJPE-dest, improving over BiFu's 903 mm and 234.2 mm.
- Removing the geometry discriminator raises Traj-dest on GIMO from 666 to 677 mm; removing PointNet++ raises it to 918 mm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SIF3D's scene-and-gaze-aware prediction instead of BiFu with scene and gaze | GIMO · GIMO | Traj-dest: 727 -> 666 | Helped |
| add SIF3D's scene-and-gaze-aware prediction instead of BiFu with scene and gaze | GTA-1M · GTA-1M | MPJPE-dest: 234.2 -> 227.7 | Helped |
| add TIA (ablation) | SIF3D · GIMO | Traj-dest: 850 -> 666 | Helped |
| add TIA (ablation) | SIF3D · GIMO | MPJPE-dest: 203.8 -> 195.7 | Helped |
| add SCA (ablation) | SIF3D · GIMO | Traj-dest: 703 -> 666 | Helped |
| add SCA (ablation) | SIF3D · GIMO | MPJPE-dest: 200.4 -> 195.7 | Helped |
| add MotionDecoder (ablation) | SIF3D · GIMO | Traj-dest: 699 -> 666 | Helped |
| add MotionDecoder (ablation) | SIF3D · GIMO | MPJPE-dest: 198.9 -> 195.7 | Helped |
| add geometry discriminator (ablation) | SIF3D · GIMO | Traj-dest: 677 -> 666 | Helped |
| add geometry discriminator (ablation) | SIF3D · GIMO | MPJPE-dest: 196.8 -> 195.7 | Helped |
| use PointNet++ instead of an MLP scene encoder (ablation) | SIF3D · GIMO | Traj-dest: 918 -> 666 | Helped |
| use PointNet++ instead of an MLP scene encoder (ablation) | SIF3D · GIMO | MPJPE-dest: 218.5 -> 195.7 | Helped |
Surprising findings
- Weighted temporal aggregators (Conv, Transformer) performed worse than simply using the last motion embedding, which the authors attribute to limited dataset size and weak supervision.
Limitations
- Performance degrades or gains are limited for some baselines; SIF3D does not uniformly win on every metric (e.g., GIMO Traj-path is worse than LTD/SPGSN without scene).
- Gaze is approximated on GTA-1M by the ray-to-face intersection, not measured.
- Scene point clouds are down-sampled to 4096 points, risking loss of critical scene detail.
Experimental setup
Datasets: GIMO GTA-1M
Benchmarks: GIMO GTA-1M
Metrics: Traj-path Traj-dest MPJPE-path MPJPE-dest
Baselines: LTD SPGSN BiFu AuxFormer
Builds on: PointNet++ BiFu VPoser SMPL-X LTD
Closest papers by meaning
- Vision-Guided Action: Enhancing 3D Human Motion Prediction with Gaze-informed Affordance in 3D Scenes (CVPR 2025 · similarity 0.88)
- MotionMap: Representing Multimodality in Human Pose Forecasting (CVPR 2025 · similarity 0.88)
- Forecasting of 3D Whole-body Human Poses with Grasping Objects (CVPR 2024 · similarity 0.87)
- Expressive Forecasting of 3D Whole-Body Human Motions (AAAI 2024 · similarity 0.87)
- Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning (CVPR 2024 · similarity 0.87)
- FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observations (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.