Research map / Label-efficient and robust learning / Recognition and long-tail

MaskCLR: Attention-Guided Contrastive Learning for Robust Action Representation Learning

CVPR 2024 · Label-efficient and robust learning › Recognition and long-tail · approach: Correspondence and causal representation learning · 20 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MaskCLR, which trains transformer skeleton action recognizers with attention-guided probabilistic joint masking plus sample- and class-level contrastive losses to improve accuracy, robustness, and cross-pose-estimator generalization.

Problem

Transformer-based skeleton action recognition relies on a small set of highly attended joints, so it degrades under small skeleton perturbations and when the pose estimator changes between training and testing.

Approach

Mask the most-attended joints (sampled from a sharpened softmax over attention scores via Gumbel top-K) and feed them to a masked pathway; align standard and masked representations at the sample level and align class-averaged representations with a class-contrastive loss that also targets confusing TP/FN/FP samples.

Key results

• MaskCLR with DSTFormer reaches 93.9% on NTU60-XSub, 97.3% on NTU60-XView, 87.4% on NTU120-XSub and 44.7% on Kinetics400, improving over MotionBERT by 5.8 points on Kinetics400.
• Trained on MQ HRNet skeletons and tested on HQ ViTPose skeletons, MaskCLR improves NTU120-XSub by 26.6 points over MotionBERT (78.3% vs 51.7%).
• Attention-guided masking at delta=0.7 reaches 91.7% on NTU60-XSub versus 89.7% for the best random masking ratio (15%).

Measured effects reported in the paper

Surprising findings

• Random masking at ratios of 0.5 or more and attention-guided masking above 0.7 both cause a rapid accuracy drop, suggesting too many masked joints leave too few cues.
• Under HRNet-to-ViTPose transfer, MotionBERT's NTU120-XSub accuracy falls to 51.7%, a large drop from its standard-setting accuracy.

Limitations

• Masked pathway doubles training forward passes, adding training cost.
• Gains on NTU60-XView are marginal (tied with MAMP on the vanilla transformer backbone).

Experimental setup

Datasets: NTU RGB+D 60 NTU RGB+D 120 Kinetics400

Benchmarks: NTU60-XSub NTU60-XView NTU120-XSub NTU120-XSet Kinetics400

Metrics: Top-1 accuracy

Baselines: MotionBERT PoseConv3D ST-GCN++ MS-G3D AAGCN CTR-GCN MAMP SkeletonMAE AimCLR CrosSCLR ActCLR FG-STFormer

Builds on: MotionBERT DSTFormer STTFormer SegFormer contrastive penalty Gumbel max trick

Closest papers by meaning

• Less is More: Compact-Token Masked Feature Learning for Skeleton Representation Learning (NeurIPS 2026 · similarity 0.89)
• Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition (CVPR 2025 · similarity 0.87)
• Stitch, Contrast, and Segment: Learning a Human Action Segmentation Model Using Trimmed Skeleton Videos (AAAI 2025 · similarity 0.87)
• Skeleton-based Action Recognition with Non-linear Dependency Modeling and Hilbert-Schmidt Independence Criterion (AAAI 2025 · similarity 0.87)
• Rethinking Masked Data Reconstruction Pretraining for Strong 3D Action Representation Learning (AAAI 2025 · similarity 0.87)
• Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition (CVPR 2025 · similarity 0.86)