Research map / Point clouds / Point cloud classification
A Unified Framework for Human-centric Point Cloud Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniPVU-Human, a unified framework for human-centric LiDAR point cloud video understanding that combines body-part segmentation and motion flow priors from synthetic data with masked body-part self-learning and hierarchical fine-tuning.
Problem
Human-centric point cloud video understanding relies on heavily labeled data and generic point cloud backbones that ignore human-specific structure and motion, leading to poor generalization across tasks such as action recognition and 3D pose estimation.
Approach
Extracts human priors (body-part segmentation via HBSeg and point-wise motion flow via HMFlow, both pre-trained on synthetic SMPL-based LiDAR data), then masks and reconstructs body-part patches with spatio-temporal transformers, and fine-tunes with global, part, and point-level features.
Key results
- On HuCenLife action recognition, UniPVU-Human reaches 61.8% mAcc versus 54.1% for MaST-Pre and 58.1% for PointMLP.
- On LIP 3D pose estimation (pure point cloud), MPJPE is 58.8 mm versus 60.1 mm for LIP; with PC+IMU it is 47.2 mm versus 48.9 mm.
- Ablation shows self-learning adds about 6.2 points of mAcc, and with 20% of fine-tuning data it reaches 51% versus 39.8% for MaST-Pre.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UniPVU-Human instead of LIP (PC) | 3D pose estimation on point clouds · LIP test (PC) | MPJPE (mm): 60.1 -> 58.8 | Helped |
| Use UniPVU-Human instead of LIP (PC+IMU) | 3D pose estimation with point clouds and IMU · LIP test (PC+IMU) | MPJPE (mm): 48.9 -> 47.2 | Helped |
| Use UniPVU-Human rather than the strongest reported action-recognition baseline | HuCenLife action recognition · HuCenLife test | mAcc: 58.1 -> 61.8 | Helped |
| Add self-learning to PointMAE (ablation) | PointMAE on HuCenLife · HuCenLife test | mAcc: 53.4% and 56.1% | Helped |
| Add hierarchical features during fine-tuning (ablation) | PointMAE on HuCenLife · HuCenLife test | mAcc: +1.9% | Helped |
| Use body-part patches instead of generic point-cloud patches (ablation) | UniPVU-Human on HuCenLife · HuCenLife test | mAcc: 55.6 -> 59.9 | Helped |
| Add both spatial and temporal masking (ablation) | UniPVU-Human on HuCenLife · HuCenLife test | mAcc: 6.2% | Helped |
| Add a global token (ablation) | UniPVU-Human on HuCenLife · HuCenLife test | mAcc: 59.3 -> 61.3 | Helped |
| Add point-wise motion flow (ablation) | UniPVU-Human on HuCenLife · HuCenLife test | mAcc: 58.9 -> 61.8 | Helped |
| Pretrain with self-learning rather than training directly (ablation) | UniPVU-Human on HuCenLife with 20% training data · HuCenLife | mAcc: 44.9(-10.9) -> 51(-10.8) | Helped |
Surprising findings
- Adding global token and motion flow to PointMAE only raises mAcc from 56.1 to 58.0, while the full model reaches 61.8.
- UniPVU-Human scores below several static baselines on some action classes such as lift (27.1 vs 53.4 for PointMAE).
Limitations
- Relies on synthetic LiDAR data generated from SMPL meshes for prior-network pretraining, with a possible sim-to-real gap.
- Evaluated only on two LiDAR datasets (HuCenLife and LIPD) and two tasks.
- Code, datasets and models are only promised ('will be released soon').
Experimental setup
Datasets: AMASS SURREAL HuCenLife LIPD SMPL
Benchmarks: HuCenLife LIPD
Metrics: mAcc MPJPE
Baselines: PointNet PointNet++ PointMLP PointNeXt PCT HuCenLife PointMAE MaST-Pre LiDARCap LIP
Builds on: Point-MAE FLOT PointNeXt MaST-Pre LIP
Closest papers by meaning
- 3DInAction: Understanding Human Actions in 3D Point Clouds (CVPR 2024 · similarity 0.88)
- Adapting Pre-trained 3D Models for Point Cloud Video Understanding via Cross-frame Spatio-temporal Perception (CVPR 2025 · similarity 0.86)
- PvNeXt: Rethinking Network Design and Temporal Motion for Point Cloud Video Recognition (ICLR 2025 · similarity 0.85)
- HUNTER: Unsupervised Human-centric 3D Detection via Transferring Knowledge from Synthetic Instances to Real Scenes (CVPR 2024 · similarity 0.85)
- RELI11D: A Comprehensive Multimodal Human Motion Dataset and Method (CVPR 2024 · similarity 0.85)
- Cross-view and Cross-pose Completion for 3D Human Understanding (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.