Research map / Label-efficient and robust learning / Self-supervised and contrastive
SCD-Net: Spatiotemporal Clues Disentanglement Network for Self-Supervised Skeleton-Based Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCD-Net, a MoCo-style contrastive framework for self-supervised skeleton action recognition that decouples spatial and temporal clues with a dual-path encoder, adds a cross-domain loss anchored on a global representation, and uses structurally constrained spatial and cube-based temporal masking.
Problem
Self-supervised skeleton-based action recognition, where existing contrastive methods encode entangled spatiotemporal representations and contrast only at the same representation level, with limited data augmentation beyond scale transformations.
Approach
A dual-path decoupling encoder (GCN feature extractor followed by spatial and temporal projection plus transformer refinement) yields explicit spatial and temporal clues; a global anchor feature is contrasted across domains with InfoNCE; and structured masking (adjacency-based spatial masking and cube-based temporal masking) augments the views.
Key results
- Linear evaluation on NTU-60 x-sub reaches 86.6% top-1 versus 81.1% for the previous best HiCo-Transformer, and PKU-MMD II x-sub reaches 54.0% versus 52.9%.
- Semi-supervised learning with 1% labels on NTU-60 x-sub achieves 69.1% versus 54.4% for HiCo-Transformer.
- Ablation on NTU-60 x-sub: removing the feature extractor drops top-1 to 80.0%, and removing the decoupling module drops it to 63.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SCD-Net instead of HiCo-Transformer | action recognition · NTU-60 x-sub | accuracy: 81.1 -> 86.6 | Helped |
| use SCD-Net instead of HiCo-Transformer | action retrieval · NTU-120 x-setup | accuracy: 59.1 -> 65.7 | Helped |
| use SCD-Net instead of HiCo-Transformer | transfer learning to PKU-MMD II from NTU-60 · PKU-MMD II | accuracy: 56.3 -> 67.5 | Helped |
| use SCD-Net instead of HiCo-Transformer | semi-supervised action recognition · NTU-60 x-view, 1% labels | accuracy: 54.8 -> 66.8 | Helped |
| remove the feature extractor (ablation) | SCD-Net · NTU-60 x-sub | Top-1 accuracy: 86.6 -> 80.0 | Hurt |
| use a shared instead of non-shared feature extractor (ablation) | SCD-Net · NTU-60 x-sub | Top-1 accuracy: 86.6 -> 85.2 | Hurt |
| remove the decoupling module (ablation) | SCD-Net · NTU-60 x-sub | Top-1 accuracy: 86.6 -> 63.7 | Hurt |
| remove refinement from decoupling (ablation) | SCD-Net · NTU-60 x-sub | Top-1 accuracy: 86.6 -> 84.0 | Hurt |
| use interactive loss instead of instance loss with S-T granularity (ablation) | SCD-Net · NTU-60 x-sub | Top-1 accuracy: 84.6 -> 85.3 | Helped |
| combine interactive and instance losses (ablation) | SCD-Net · NTU-60 x-sub | Top-1 accuracy: 86.6 -> 86.5 | No clear effect |
| add structured spatial and temporal masking to conventional augmentation (ablation) | SCD-Net · NTU-60 x-sub | Top-1 accuracy: 85.4 -> 86.6 | Helped |
| replace structured masking with random spatiotemporal masking (ablation) | SCD-Net · NTU-60 x-sub | Top-1 accuracy: 86.6 -> 85.6 | Hurt |
Surprising findings
- Completely isolating spatial and temporal information (no feature extractor) lowers accuracy, so shared spatiotemporal modelling is reported as necessary.
- Using masking alone without conventional augmentation performs far worse than conventional augmentation alone (76.0% vs 85.4%).
Limitations
- Using both interactive and instance losses together does not improve results, so the benefit of combining guidance signals is unclear.
- Performance is reported as relatively insensitive to architecture size, but sensitivity to masking hyperparameters is only partially explored.
Experimental setup
Datasets: NTU-RGB+D 60 NTU-RGB+D 120 PKU-MMD I PKU-MMD II
Benchmarks: NTU-RGB+D 60 NTU-RGB+D 120 PKU-MMD I PKU-MMD II
Metrics: top-1 accuracy top-5 accuracy
Baselines: HiCo-Transformer SkeAttnCLR ActCLR AimCLR ISC CrosSCLR GL-Transformer HYSP Colorization LongT GAN
Builds on: MoCo CTR-GCN InfoNCE HiCo-Transformer CrosSCLR
Closest papers by meaning
- Graph Contrastive Learning for Skeleton-based Action Recognition (ICLR 2023 · similarity 0.88)
- Self-Supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences (AAAI 2023 · similarity 0.88)
- Towards To-a-T Spatio-Temporal Focus for Skeleton-Based Action Recognition (AAAI 2022 · similarity 0.87)
- Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition (CVPR 2025 · similarity 0.86)
- MaskCLR: Attention-Guided Contrastive Learning for Robust Action Representation Learning (CVPR 2024 · similarity 0.85)
- Prompt-guided Disentangled Representation for Action Recognition (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.