Research map / Label-efficient and robust learning / Recognition and long-tail
USDRL: Unified Skeleton-Based Dense Representation Learning with Multi-Grained Feature Decorrelation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes USDRL, a non-negative (feature decorrelation) self-supervised framework for skeleton dense representation learning, with multi-grained decorrelation losses over temporal, spatial, and instance domains and a Transformer-based DSTE encoder.
Problem
Self-supervised skeleton-based action representation learning mostly relies on negative-sample contrastive learning (needing momentum encoders/memory banks) and learns global instance-level features that transfer poorly to dense prediction tasks such as action detection.
Approach
Applies feature decorrelation (Barlow Twins/VICReg-style variance, covariance and cross-correlation terms plus intra-sample consistency) at temporal, spatial and instance granularity, combined with a DSTE encoder (DenseShift attention plus convolutional attention) that preserves per-frame and per-joint representations.
Key results
- USDRL (DSTE) reaches 85.2% on NTU-60 x-sub and 91.7% on x-view for unsupervised recognition with joint modality only, beating UmURL (3 modalities) at 84.2% and 90.9%.
- On PKU-MMD I action detection, USDRL (DSTE) achieves 75.7 mAP_a versus 61.8 for PCM3, and outperforms its STTR variant by nearly 9 points.
- With 1% labels on NTU-60 x-sub, USDRL (DSTE) reaches 57.3% versus 53.8% for PCM3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the XC matrix (ablation) | USDRL · NTU-60 recognition and retrieval | accuracy: +0.5% | Helped |
| Add multi-grained feature decorrelation (MG-FD) (ablation) | DSTE · NTU-60 xsub recognition | accuracy: 84.6 -> 85.2 | Helped |
| Add multi-grained feature decorrelation (MG-FD) (ablation) | DSTE · NTU-60 xsub retrieval | accuracy: 74.7 -> 75.0 | Helped |
| Use feature decorrelation rather than the negative-based pretraining baseline (ablation) | USDRL · NTU-60 recognition and retrieval | accuracy: 74.8 -> 83.7 (recognition); 63.8 -> 73.2 (retrieval) | Helped |
| Use both CA and DSA rather than either module alone (ablation) | DSTE · NTU-60 xsub action recognition | accuracy | Helped |
| Use DSTE instead of STTR | USDRL · PKU-MMD I xsub action detection | mAP_a (%): 66.1 -> 75.7 | Helped |
| Use DSTE instead of STTR | USDRL · PKU-MMD I xsub action detection | mAP_v (%): 65.9 -> 74.9 | Helped |
Surprising findings
- Replacing the DSTE encoder with the standard STTR Transformer lowers results but USDRL with STTR still beats most prior feature-decorrelation methods, indicating the loss contributes independently of the encoder.
Limitations
- Gap to negative-based methods and multi-modal ensembles remains on some protocols; 3-stream ensemble required for best NTU-120 results.
- Action detection evaluated only on PKU-MMD I; pretraining on NTU-60 x-sub only for detection transfer.
Experimental setup
Datasets: NTU-RGB+D 60 NTU-RGB+D 120 PKU-MMD I PKU-MMD II
Benchmarks: NTU-RGB+D 60 NTU-RGB+D 120 PKU-MMD I PKU-MMD II
Metrics: top-1 accuracy retrieval accuracy mAP (action and video)
Baselines: MS2L ViA PCM3 GL-Transformer Masked Colorization MAMP AimCLR CMD HaLP HiCo HYSP UmURL
Builds on: Barlow Twins VICReg W-MSE STTR
Closest papers by meaning
- Less is More: Compact-Token Masked Feature Learning for Skeleton Representation Learning (NeurIPS 2026 · similarity 0.87)
- Part-aware Unified Representation of Language and Skeleton for Zero-shot Action Recognition (CVPR 2024 · similarity 0.86)
- Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition (CVPR 2025 · similarity 0.85)
- Heterogeneous Skeleton-Based Action Representation Learning (CVPR 2025 · similarity 0.85)
- MaskCLR: Attention-Guided Contrastive Learning for Robust Action Representation Learning (CVPR 2024 · similarity 0.85)
- Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.