Research map / Point clouds / Point cloud classification
DeSPITE: Exploring Contrastive Deep Skeleton-Pointcloud-IMU-Text Embeddings for Advanced Point Cloud Human Activity Understanding
Thomas Kreutz, Max Mühlhäuser, Alejandro Sanchez Guinea
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add text embedding to point cloud+skeleton+IMU pre-training (DeSPIE to DeSPITE) (ablation) | DeSPITE · Babel-LIPD-v2-CLS (skeleton, fine-tuning) | Acc(Seg) (%): 68.31 -> 70.64 | Improved |
| add text embedding to pre-training (DeSPIE to DeSPITE) (ablation) | DeSPITE · Babel-LIPD-v2-CLS (point cloud, fine-tuning) | Acc(Seg) (%): 67.51 -> 69.00 | Improved |
| add text embedding to pre-training (DeSPIE to DeSPITE) (ablation) | DeSPITE · Babel-LIPD-v2-CLS (IMU, fine-tuning) | Acc(Seg) (%): 69.21 -> 68.40 | Hurt |
| train with text (DeSPITE variants) instead of modalities only for matching (ablation) | DeSPITE · LIPD matching and temporal moment retrieval | accuracy / recall | Hurt |
| pre-train with DeSPITE then fine-tune PST-Transformer vs supervised | PST-Transformer · HMPEAR | Acc(Seg) (%): 65.94 -> 69.18 | Improved |
| pre-train with DePITE then fine-tune PST-Transformer | PST-Transformer · HMPEAR | Acc(Seg) (%): 65.94 -> 70.65 | Improved |
| pre-train with DeSPIE then fine-tune PST-Transformer | PST-Transformer · MSR-Action3D | clip accuracy Acc@1 (%): 92.33 -> 95.47 | Improved |
| multi-modal pre-training vs random init with fine-tuning | ACTOR skeleton encoder · Babel-LIPD-v2-CLS | Acc(Seg) (%): 68.23 -> 70.64 | Improved |
| multi-modal pre-training vs random init with fine-tuning | LSTM IMU encoder · Babel-LIPD-v2-CLS | Acc(Seg) (%): 65.62 -> 69.21 | Improved |
| add IMU to point cloud pre-training before fine-tuning (PIE vs random init, non-linear) (ablation) | PST-Transformer · Babel-LIPD-v2-CLS (point cloud) | Acc(Seg) (%): 67.38 -> 69.52 | Improved |
| add skeleton to point cloud pre-training (SPE) before fine-tuning (ablation) | PST-Transformer · Babel-LIPD-v2-CLS (point cloud) | Acc(Seg) (%): 67.38 -> 66.13 | Hurt |
| linear probing vs fine-tuning for skeleton encoder pre-trained with SIE (ablation) | ACTOR · Babel-LIPD-v2-CLS (skeleton) | Acc(Seg) (%): 70.44 -> 57.14 | Hurt |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 2 of 2 on most shared results.
- MAMBA4D — Mamba4D: Efficient 4D Point Cloud Video Understanding with Disentangled Spatial-Temporal State Space Models (CVPR 2025) · better on 1 of 1 shared result
- PvNext — PvNeXt: Rethinking Network Design and Temporal Motion for Point Cloud Video Recognition (ICLR 2025) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- A Unified Framework for Human-centric Point Cloud Video Understanding (CVPR 2024 · similarity 0.85)
- HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors (AAAI 2024 · similarity 0.84)
- RELI11D: A Comprehensive Multimodal Human Motion Dataset and Method (CVPR 2024 · similarity 0.84)
- 3DInAction: Understanding Human Actions in 3D Point Clouds (CVPR 2024 · similarity 0.84)
- SKI Models: Skeleton Induced Vision-Language Embeddings for Understanding Activities of Daily Living (AAAI 2025 · similarity 0.84)
- Pre-training a Density-Aware Pose Transformer for Robust LiDAR-based 3D Human Pose Estimation (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
