Research map / 3D vision / Human pose estimation
RELI11D: A Comprehensive Multimodal Human Motion Dataset and Method
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RELI11D, a multimodal (LiDAR, IMU, RGB, event) sports human-motion dataset with SMPL annotations, and LEIR, a cross-attention fusion baseline for 3D/global human pose estimation.
Problem
Human pose estimation for complex, fast motions and global localization is hard with RGB, LiDAR, or IMU alone, and no public dataset combined all four modalities (RGB, LiDAR, IMU, event).
Approach
A four-modality dataset with annotations produced by fusing IMU motion capture with LiDAR point clouds and scanned scene meshes under contact, smoothness and geometry losses, plus a baseline (LEIR) that fuses LiDAR, RGB and event features via multimodal cross-attention in a temporal unified model.
Key results
- LEIR with LiDAR+RGB+Event inputs reaches MPJPE 49.19 mm and PA-MPJPE 40.87 mm on RELI11D, better than the single-modality LiDAR-only (59.93 mm) and RGB-only (62.71 mm) variants of LEIR.
- On LiDARHuman26M, LEIR with LiDAR+RGB reaches MPJPE 75.09 mm versus 78.18 mm for FusionPose and 96.93 mm for ImmFusion.
- On RELI11D, image-based 3D HPE methods such as HybrIK, NIKI and SMPLer-X achieve MPJPE of 249.34, 196.68 and 171.97 mm, showing the benchmark is challenging.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LiDAR, RGB, and Event inputs instead of RGB alone (ablation) | LEIR · RELI11D | MPJPE: 62.71 -> 49.19 | Helped |
| Add RGB and Event inputs to LiDAR (ablation) | LEIR · RELI11D | MPJPE: 59.93 -> 49.19 | Helped |
| Add RGB input to LiDAR (ablation) | LEIR · RELI11D | MPJPE: 59.93 -> 55.36 | Helped |
| Add Event input to LiDAR (ablation) | LEIR · RELI11D | MPJPE: 59.93 -> 57.79 | Helped |
| Use LiDAR, RGB, and Event inputs instead of RGB alone (ablation) | LEIR · RELI11D | T-Error: 710.84 -> 146.13 | Helped |
| Use LiDAR, RGB, and Event inputs instead of Event alone (ablation) | LEIR · RELI11D | MPJPE: 107.78 -> 49.19 | Helped |
| Add Lcontact, Lsmoo, and Lgeo to the optimization (ablation) | RELI11D annotation optimization · PingPong1 | MPJPE: 29.14 -> 12.25 | Helped |
| Add Lcontact, Lsmoo, and Lgeo to the optimization (ablation) | RELI11D annotation optimization · Boxing1 | MPJPE: 31.20 -> 10.26 | Helped |
| Add Lcontact (ablation) | RELI11D annotation optimization · PingPong1 | MPJPE: 13.75 -> 12.25 | Helped |
| Add Lsmoo (ablation) | RELI11D annotation optimization · PingPong1 | MPJPE: 15.19 -> 12.25 | Helped |
| Add Lgeo (ablation) | RELI11D annotation optimization · PingPong1 | MPJPE: 18.14 -> 12.25 | Helped |
| Add Lcontact (ablation) | RELI11D annotation optimization · Boxing1 | MPJPE: 11.89 -> 10.26 | Helped |
Surprising findings
- Event-only input gave the weakest LEIR results (MPJPE 107.78 mm), and the RGB-only global GMPJPE was very large (557.81 mm), so monocular RGB fails at global localization without LiDAR.
Limitations
- Dataset is limited to 10 actors, 5 sports and 7 scenes, with 48 sequences.
- Existing HPE methods perform poorly on RELI11D, and LEIR's gains over baselines are modest on LiDARHuman26M.
- IMU translation is inaccurate, so global trajectories depend on LiDAR-based hip localization and optimization.
Experimental setup
Datasets: RELI11D LiDARHuman26M
Benchmarks: RELI11D LiDARHuman26M
Metrics: MPJPE PA-MPJPE PCK0.3 PVE ACCEL GMPJPE T-Error
Baselines: LiDARCap P4Transformer PCT HybrIK NIKI SMPLer-X EventHPE EventPointPose ImmFusion FusionPose GLAMR TRACE
Builds on: LiDARCap DINOv2 PointNet++ SMPL ST-GCN
Closest papers by meaning
- LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation in Free Environment (CVPR 2024 · similarity 0.87)
- I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions (CVPR 2024 · similarity 0.85)
- Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for Scene Flow (CVPR 2024 · similarity 0.85)
- Motions as Queries: One-Stage Multi-Person Holistic Human Motion Capture (CVPR 2025 · similarity 0.85)
- MotionMap: Representing Multimodality in Human Pose Forecasting (CVPR 2025 · similarity 0.85)
- A Unified Framework for Human-centric Point Cloud Video Understanding (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.