Research map / 3D vision / Human pose estimation
Spatial-Related Sensors Matters: 3D Human Motion Reconstruction Assisted with Textual Semantics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a text-supervised sparse-IMU 3D human motion reconstruction method using uncertainty-guided spatial attention, a Hierarchical Temporal Transformer, and text-sensor contrastive learning, evaluated on Totalcapture and DIP-IMU.
Problem
Reconstructing 3D human motion from sparse IMUs is under-constrained because different poses yield identical sensor readings (e.g., sitting vs. standing).
Approach
Model intra-frame spatial relations between IMUs with attention scaled by per-sensor predicted uncertainty (with uncertainty-guided resampling), and resolve pose ambiguity by aligning sensor sequences with textual action labels via contrastive learning and cross-attention fusion.
Key results
- Offline Totalcapture: SIP error 7.92 deg vs 12.30 for Transpose; positional error 3.70 cm vs 4.85 cm.
- Online Totalcapture: SIP error 9.67 deg vs 11.74 for TIP; mesh error 5.05 cm vs 6.10 for TIP.
- Ablation: removing text raises Totalcapture SIP error from 7.92 to 9.21 deg.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add text semantic information (ablation) | Ours · Totalcapture offline | SIP Err (deg): 9.21(+/-4.75) -> 7.92(+/-4.38) | Helped |
| add text semantic information (ablation) | Ours · Totalcapture offline | Ang Err (deg): 10.30(+/-4.43) -> 9.35(+/-4.10) | Helped |
| add text semantic information (ablation) | Ours · Totalcapture offline | Pos Err (cm): 4.19(+/-2.23) -> 3.70(+/-2.03) | Helped |
| add UGSA (ablation) | Ours · Totalcapture offline | SIP Err (deg): 8.67(+/-4.73) -> 7.92(+/-4.38) | Helped |
| add UGSA (ablation) | Ours · Totalcapture offline | Ang Err (deg): 9.94(+/-4.37) -> 9.35(+/-4.10) | Helped |
| add UGSA (ablation) | Ours · Totalcapture offline | Pos Err (cm): 4.04(+/-2.22) -> 3.70(+/-2.03) | Helped |
| add HTT (ablation) | Ours · Totalcapture offline | SIP Err (deg): 8.35(+/-4.57) -> 7.92(+/-4.38) | Helped |
| add HTT (ablation) | Ours · Totalcapture offline | Ang Err (deg): 9.70(+/-4.29) -> 9.35(+/-4.10) | Helped |
| add HTT (ablation) | Ours · Totalcapture offline | Jitter: 0.44(+/-1.21) -> 1.74($\pm$1.55) | Hurt |
Surprising findings
- Jitter is worse than baselines on Totalcapture (1.74 vs 1.31 offline), which the authors attribute to the sliding window and patch merging, and argue jitter is less critical.
Limitations
- Full model requires semantic text annotations and cannot run in real time.
- Weaker performance on DIP-IMU due to fewer and less detailed semantic labels.
- Jitter metric is not as robust as pose-accuracy metrics.
Experimental setup
Datasets: AMASS DIP-IMU Totalcapture Babel
Benchmarks: Totalcapture DIP-IMU
Metrics: SIP error angular error positional error mesh error jitter
Baselines: SIP DIP Transpose PIP TIP
Builds on: CLIP Swin Transformer Transpose DIP
Closest papers by meaning
- DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion (CVPR 2024 · similarity 0.86)
- UMotion: Uncertainty-driven Human Motion Estimation from Inertial and Ultra-wideband Units (CVPR 2025 · similarity 0.85)
- Inertia-1: An Open Exploration of Wearable Motion Foundation Models (NeurIPS 2026 · similarity 0.85)
- Dynamic Inertial Poser (DynaIP): Part-Based Motion Dynamics Learning for Enhanced Human Pose Estimation with Sparse Inertial Sensors (CVPR 2024 · similarity 0.85)
- MoPFormer: Motion-Primitive Transformer for Wearable-Sensor Activity Recognition (NeurIPS 2025 · similarity 0.85)
- Full-Body Motion Reconstruction with Sparse Sensing from Graph Perspective (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.