Research map / Label-efficient and robust learning / Recognition and long-tail
Heterogeneous Skeleton-Based Action Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a unified self-supervised framework that processes heterogeneous 25-joint 3D and 17-joint 2D skeletons (via 3D lifting, prompted unified skeleton, and CLIP-based semantic motion encoding) with a shared Transformer trained with VICReg-style losses.
Problem
Skeleton-based action representation learning assumes homogeneous skeletons, ignoring heterogeneity in joint count and coordinate dimensionality (e.g., 3D 25-joint Kinect vs. 2D 17-joint pose estimates).
Approach
Unify heterogeneous skeletons by lifting 2D 17-joint poses to 3D with an auxiliary network, padding each skeleton to a common 30-joint topology using trainable skeleton-specific prompts, adding a semantic motion encoding from text embeddings of motion direction words, and learning a shared Transformer with feature-consistency and VICReg regularization.
Key results
- Unified J+C+S model reaches 87.8% on NTU-60 x-sub and 93.7% on NTU-120 x-sub under linear evaluation, above the 3s-UmURL baseline (84.4% / 91.4%).
- On PKU-MMD II x-sub the J+C+S model reaches 58.2%, versus 54.3% for 3s-UmURL.
- Transfer from NTU-120 to FineGYM yields 75.3%, the best among compared methods in Table 6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the unified J + C + S representation instead of the strongest listed baseline | Action recognition · NTU-60 x-sub | top-1 accuracy: 87.1 -> 87.8 | Helped |
| Use the unified J + C + S representation instead of the strongest listed baseline | Action recognition · NTU-60 x-view | top-1 accuracy: 93.2 -> 93.7 | Helped |
| Use the unified J + C + S representation instead of the strongest listed baseline | Action recognition · NTU-120 x-sub | top-1 accuracy: 80.0 -> 78.9 | Hurt |
| Use the unified J + C + S representation instead of the strongest listed baseline | Action recognition · NTU-120 x-set | top-1 accuracy: 81.4 -> 82.2 | Helped |
| Use the unified J + C + S representation instead of the strongest listed baseline | Action recognition · PKU-MMD II x-sub | top-1 accuracy: 59.7 -> 58.2 | Hurt |
| Use the unified J + C + S representation instead of the strongest listed baseline | Action retrieval · NTU-60 x-sub | accuracy: 72.0 -> 72.7 | Helped |
| Use the unified J + C + S representation instead of the strongest listed baseline | Action retrieval · NTU-60 x-view | accuracy: 88.9 -> 90.9 | Helped |
| Use the unified J + C + S representation instead of the strongest listed baseline | Action retrieval · NTU-120 x-sub | accuracy: 59.5 -> 61.9 | Helped |
| Use the unified J + C + S representation instead of the strongest listed baseline | Action retrieval · NTU-120 x-set | accuracy: 62.2 -> 66.9 | Helped |
| Add 3D pose estimation to reconstruct the 2D skeleton as 3D (ablation) | Ours · PKU-MMD II x-sub | top-1 accuracy: 55.8 -> 58.2 | Helped |
| Add language-model semantic motion encoding instead of numeric joint-motion codes (ablation) | Ours · PKU-MMD II x-sub | top-1 accuracy: 57.9 -> 58.2 | Helped |
| Add trainable skeleton-specific prompts instead of zero-padding (ablation) | Ours · PKU-MMD II x-sub | top-1 accuracy: 57.2 -> 58.2 | Helped |
Surprising findings
- Semantic-motion-only encoding (S) gives weaker accuracy alone (70.1% NTU-60 x-sub), but combining modalities gives large gains.
- Semi-supervised results are uneven: the method beats baselines at 5% labels but only gets relatively good results at 1% labels.
Limitations
- Model is limited to processing skeleton data for up to two people.
- Only two skeleton types (25-joint 3D and 17-joint 2D) are studied.
- Performance is weak with little labeled data in semi-supervised setting (1%).
Experimental setup
Datasets: NTU-60 NTU-120 PKU-MMD II FineGYM
Benchmarks: NTU-60 NTU-120 PKU-MMD II FineGYM
Metrics: top-1 accuracy FLOPs
Baselines: AimCLR PTSL GL-Transformer CPM CMD IGM UmURL HaLP ActCLR USDRL CrosSCLR HiCo
Builds on: VICReg UmURL CrosSCLR CLIP
Closest papers by meaning
- Skeleton-based Action Recognition with Non-linear Dependency Modeling and Hilbert-Schmidt Independence Criterion (AAAI 2025 · similarity 0.91)
- Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition (CVPR 2025 · similarity 0.90)
- Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition (CVPR 2025 · similarity 0.90)
- SAMoR: Motion Modelling for Articulated Objects of Any Skeleton and Topology (NeurIPS 2026 · similarity 0.90)
- Part-aware Unified Representation of Language and Skeleton for Zero-shot Action Recognition (CVPR 2024 · similarity 0.89)
- Stitch, Contrast, and Segment: Learning a Human Action Segmentation Model Using Trimmed Skeleton Videos (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.