Research map / Video / Action recognition
Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProtoGCN, a GCN-based skeleton action recognizer that reconstructs features from learnable memory prototypes of joint relationships, with motion topology enhancement and class-specific contrastive learning, reaching SOTA on NTU-60/120, Kinetics-Skeleton and FineGYM.
Problem
Distinguishing similar skeleton-based actions whose joint trajectories look alike, since skeleton representations lack image-level detail and current GCNs capture only coarse motion patterns.
Approach
A Prototype Reconstruction Network stores learnable relationship prototypes in a memory and rebuilds each sample's graph topology as an attention-weighted combination of them, trained with a class-specific contrastive loss and augmented by a motion topology enhancement module that adds intra-sample and inter-sample topology terms.
Key results
- ProtoGCN (6-ensemble) reaches 93.8% X-Sub and 97.8% X-View on NTU RGB+D, and 90.9% X-Sub and 92.2% X-Set on NTU-120.
- On FineGYM, ProtoGCN achieves 95.9% versus 94.3% for PoseConv3D and 94.1% for PYSKL.
- Single-stream ablation on NTU-120 X-Sub (bone) improves the baseline from 87.8% to 89.0%, with PRN contributing +0.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ProtoGCN (6-ensemble) instead of BlockGCN | Skeleton-based action recognition · NTU RGB+D X-Sub | accuracy: 93.1 -> 93.8 | Helped |
| use ProtoGCN (6-ensemble) instead of JT-GraphFormer | Skeleton-based action recognition · NTU RGB+D X-View | accuracy: 97.5 -> 97.8 | Helped |
| use ProtoGCN (6-ensemble) instead of BlockGCN | Skeleton-based action recognition · NTU RGB+D 120 X-Sub | accuracy: 90.3 -> 90.9 | Helped |
| use ProtoGCN (6-ensemble) instead of JT-GraphFormer | Skeleton-based action recognition · NTU RGB+D 120 X-Set | accuracy: 91.7 -> 92.2 | Helped |
| use ProtoGCN (6-ensemble) instead of DS-GCN | Skeleton-based action recognition · Kinetics-Skeleton | Top-1 accuracy: 50.6 -> 51.9 | Helped |
| use ProtoGCN instead of PoseConv3D | Skeleton-based action recognition · FineGYM | mean class Top-1 accuracy: 94.3 -> 95.9 | Helped |
| add Motion Topology Enhancement (MTE) (ablation) | PYSKL baseline · NTU RGB+D 120 X-Sub (bone modality) | accuracy: 87.8 -> 88.1 | Helped |
| add Class-Specific Contrastive Learning (CSCL) (ablation) | PYSKL baseline with MTE · NTU RGB+D 120 X-Sub (bone modality) | accuracy: 88.1 -> 88.3 | Helped |
| add Prototype Reconstruction Network (PRN) (ablation) | PYSKL baseline with MTE and CSCL · NTU RGB+D 120 X-Sub (bone modality) | accuracy: 88.3 -> 89.0 | Helped |
| replace the proposed contrastive approach with SkeletonGCL (ablation) | PYSKL baseline · NTU RGB+D 120 X-Sub (bone modality) | accuracy: 89.0 -> 88.0 | Hurt |
| replace the proposed contrastive approach with FR-Head (ablation) | PYSKL baseline · NTU RGB+D 120 X-Sub (bone modality) | accuracy: 89.0 -> 88.2 | Hurt |
| remove Aintra from the enhanced topology (ablation) | ProtoGCN · NTU RGB+D 120 X-Sub (bone modality) | accuracy: 88.8 -> 89.0 | Hurt |
Limitations
- Gains over strong baselines on NTU-60 are small (about 0.7-0.8%) and within typical run-to-run variance.
- Results on the largest gains rely on a six-stream ensemble strategy for the headline numbers.
Experimental setup
Datasets: NTU RGB+D NTU RGB+D 120 Kinetics-Skeleton FineGYM
Benchmarks: NTU RGB+D NTU RGB+D 120 Kinetics-Skeleton FineGYM
Metrics: top-1 accuracy top-5 accuracy mean class top-1 accuracy
Baselines: ST-GCN AS-GCN 2s-AGCN MS-G3D CTR-GCN InfoGCN PYSKL SkeletonGCL FR-Head HD-GCN BlockGCN DS-GCN
Builds on: PYSKL CTR-GCN InfoGCN SkeletonGCL FR-Head
Closest papers by meaning
- InfoGCN: Representation Learning for Human Skeleton-Based Action Recognition (CVPR 2022 · similarity 0.93)
- Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition (AAAI 2018 · similarity 0.92)
- BlockGCN: Redefine Topology Awareness for Skeleton-Based Action Recognition (CVPR 2024 · similarity 0.92)
- Dynamic Semantic-Based Spatial Graph Convolution Network for Skeleton-Based Human Action Recognition (AAAI 2024 · similarity 0.91)
- Heterogeneous Skeleton-Based Action Representation Learning (CVPR 2025 · similarity 0.90)
- Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.