Research map / Label-efficient and robust learning / Self-supervised and contrastive
Rethinking Masked Data Reconstruction Pretraining for Strong 3D Action Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that masked joint reconstruction for 3D skeleton self-supervised pretraining matches masked motion reconstruction when targets are normalized among timestamps (NAT), and adds a momentum-encoder InfoNCE contrastive loss to further improve representations.
Problem
Self-supervised pretraining for 3D skeleton-based action recognition: masked joint reconstruction was reported to underperform masked motion reconstruction, and masked reconstruction alone learns local rather than inter-sample relations.
Approach
Attributes the poor performance of masked joint reconstruction to segment-wise normalization that makes targets similar across timestamps, and fixes it with Normalization Among Timestamps per coordinate axis; combines the reconstruction loss with a MoCo-style momentum-encoder contrastive loss on the full skeleton.
Key results
- Linear evaluation on NTU-60 X-sub reaches 86.9% and NTU-120 X-set 80.0%, outperforming MAMP (84.9%, 78.6%) and other joint-only methods.
- With 1% labeled NTU-60 data, semi-supervised X-sub accuracy is 78.2% vs. 66.0% for MAMP.
- Joint reconstruction with NAT improves NTU-60 X-sub linear accuracy from 74.8% to 85.1%, matching masked motion reconstruction (84.9%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply NAT to the masked joint reconstruction target instead of using the raw joint target (ablation) | Masked joint reconstruction without contrastive learning · NTU-60 X-sub | top-1 accuracy: 74.8 -> 85.1 | Helped |
| Apply NAT to the masked joint reconstruction target instead of using the raw joint target (ablation) | Masked joint reconstruction without contrastive learning · NTU-120 X-sub | top-1 accuracy: 72.5 -> 78.8 | Helped |
| Use masked joint reconstruction with NAT instead of masked motion reconstruction (ablation) | Masked reconstruction without contrastive learning · NTU-60 X-sub | top-1 accuracy: 84.9 -> 85.1 | No clear effect |
| Use masked joint reconstruction with NAT instead of masked motion reconstruction (ablation) | Masked reconstruction without contrastive learning · NTU-120 X-sub | top-1 accuracy: 78.6 -> 78.8 | No clear effect |
| Add contrastive learning (ablation) | Ours with masked joint reconstruction and NAT · NTU-60 X-sub | top-1 accuracy: 85.1 -> 86.9 | Helped |
| Add contrastive learning (ablation) | Ours with masked joint reconstruction and NAT · NTU-120 X-sub | top-1 accuracy: 78.8 -> 80.0 | Helped |
| Add contrastive learning (ablation) | Ours with masked joint reconstruction and NAT · NTU-60 X-sub, 1% labeled data | top-1 accuracy: 66.2 -> 78.2 | Helped |
| Add contrastive learning (ablation) | Ours with masked joint reconstruction and NAT · NTU-60 X-view, 1% labeled data | top-1 accuracy: 68.8 -> 81.3 | Helped |
| Use Ours instead of MAMP | Linear evaluation · NTU-60 X-sub | top-1 accuracy: 84.9 -> 86.9 | Helped |
| Use Ours instead of MAMP | Linear evaluation · NTU-120 X-set | top-1 accuracy: 79.1 -> 81.5 | Helped |
| Use Ours instead of MAMP | Semi-supervised evaluation with 1% labels · NTU-60 X-sub | top-1 accuracy: +12.2% | Helped |
| Use Ours instead of MAMP | Semi-supervised evaluation with 1% labels · NTU-60 X-view | top-1 accuracy: +12.6% | Helped |
Surprising findings
- Masked joint reconstruction, previously deemed inferior, matches masked motion reconstruction once the normalization target is changed to per-axis normalization among timestamps.
- Contrastive learning gives around 12-point gains in the 1% semi-supervised setting but only about 1-2 points under full linear evaluation.
Limitations
- Contrastive learning adds only modest gains in full-data linear evaluation (1.8% and 1.2%).
- Results are reported for joint-stream only; multi-stream ensembles from prior work are not compared directly in linear evaluation.
Experimental setup
Datasets: NTU-RGB+D 60 NTU-RGB+D 120 PKU-MMD II
Benchmarks: NTU-RGB+D 60 NTU-RGB+D 120 PKU-MMD II
Metrics: top-1 accuracy
Baselines: MAMP SkeletonMAE HaLP ActCLR CMD AimCLR CrosSCLR SkateFormer SkeleTR LLM-AR
Builds on: MAMP MAE MoCo SkeletonMAE InfoNCE
Closest papers by meaning
- MaskCLR: Attention-Guided Contrastive Learning for Robust Action Representation Learning (CVPR 2024 · similarity 0.87)
- Less is More: Compact-Token Masked Feature Learning for Skeleton Representation Learning (NeurIPS 2026 · similarity 0.84)
- A Unified Masked Autoencoder with Patchified Skeletons for Motion Synthesis (AAAI 2024 · similarity 0.84)
- Video Language Model Pretraining with Spatio-temporal Masking (CVPR 2025 · similarity 0.83)
- Learning Implicit Representation for Reconstructing Articulated Objects (ICLR 2024 · similarity 0.83)
- Cross-view and Cross-pose Completion for 3D Human Understanding (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.