Research map / Vision-language and multimodal / Cross-modal learning
Navigating Open Set Scenarios for Skeleton-Based Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds an open-set skeleton-based action recognition (OS-SAR) benchmark on NTU60, NTU120 and ToyotaSmartHome with CTR-GCN, HD-GCN and Hyperformer backbones, and proposes CrossMax (cross-modality mean max discrepancy training plus CNE-distance logit refinement at test time).
Problem
Open-set recognition for skeleton-based human action recognition is unexplored; existing image and video open-set methods generalize poorly to sparse body-pose sequences and across GCN/transformer backbones, with overconfident SoftMax open-set probabilities.
Approach
Train three skeleton-stream branches (joints, bones, velocities) with a kernel-based cross-modality MMD loss to align latent spaces, then at test time refine the averaged logits using a channel-normalized Euclidean distance to the nearest training embeddings, applying different refinement to salient and non-salient logits so that open-set probabilities are better separated while closed-set accurac
Key results
- On NTU60 cross-subject with CTR-GCN, CrossMax reaches 90.62 O-AUROC and 80.32 O-AUPR versus 83.68 and 67.37 for SoftMax, with C-ACC 93.68.
- CrossMax achieves best O-AUROC across all three datasets, both evaluation protocols and all three backbones in the reported tables (e.g., 89.57 vs 81.52 SoftMax on NTU60 CS with HD-GCN).
- Ablation on NTU60 CS with CTR-GCN: Ensemble 86.23 O-AUROC, CrossMMD 88.31, CrossMax 90.62.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CrossMax instead of ARPL | CTR-GCN · NTU60 cross-subject | O-AUROC: 84.13 -> 90.62 | Helped |
| Add CrossMMD during training (ablation) | Three-modality ensemble with vanilla SoftMax · NTU60 cross-subject, CTR-GCN | O-AUROC: 86.23 -> 88.31 | Helped |
| Add CrossMMD during training (ablation) | Three-modality ensemble with vanilla SoftMax · NTU60 cross-subject, CTR-GCN | O-AUPR: 71.35 -> 74.80 | Helped |
| Add CrossMMD during training (ablation) | Three-modality ensemble with vanilla SoftMax · NTU60 cross-subject, CTR-GCN | C-ACC: 93.31 -> 93.68 | Helped |
| Add cross-modality distance-based logits refinement at test time (ablation) | CrossMMD · NTU60 cross-subject, CTR-GCN | O-AUROC: 88.31 -> 90.62 | Helped |
| Add cross-modality distance-based logits refinement at test time (ablation) | CrossMMD · NTU60 cross-subject, CTR-GCN | O-AUPR: 74.80 -> 80.32 | Helped |
| Add cross-modality distance-based logits refinement at test time (ablation) | CrossMMD · NTU60 cross-subject, CTR-GCN | C-ACC: 93.68 -> 93.68 | No clear effect |
| Replace vanilla SoftMax open-set scoring with CNE-distance (ablation) | Individual joints, bones, and velocities modalities · NTU60 | O-AUROC: +2.86%, +3.78%, and +1.12% O-AUROC | Helped |
| Use CNE-distance rather than vanilla SoftMax for class decisions (ablation) | Individual modalities · NTU60 | C-ACC | Hurt |
| Add Gaussian noise disturbance (ablation) | CrossMax · NTU60, CTR-GCN, cross-subject | O-AUROC: 90.62 -> 79.94 | Hurt |
| Add random occlusions (ablation) | CrossMax · NTU60, CTR-GCN, cross-subject | O-AUROC: 90.62 -> 84.44 | Hurt |
Surprising findings
- Existing open-set methods from image and video domains (RPL, ARPL, PMAL, DEAR, Humpty, MCD-V) do not consistently beat the SoftMax baseline across datasets and backbones; e.g., RPL and ARPL fall below SoftMax on Hyperformer for NTU60.
- The CNE-distance alone improves open-set detection but degrades closed-set accuracy relative to SoftMax, motivating the logits refinement.
Limitations
- Relies on three ensembled modality branches, tripling the model cost relative to a single stream.
- Open-set performance in some settings (e.g., ToyotaSmartHome) remains variable and baselines fluctuate across backbones and datasets.
Experimental setup
Datasets: NTU60 NTU120 ToyotaSmartHome
Benchmarks: NTU60 NTU120 ToyotaSmartHome
Metrics: O-AUROC O-AUPR C-ACC
Baselines: SoftMax RPL ARPL PMAL DEAR Humpty Dumpty MCD-V
Builds on: CTR-GCN HD-GCN Hyperformer PMAL ARPL DEAR
Closest papers by meaning
- Recovering Complete Actions for Cross-dataset Skeleton Action Recognition (NeurIPS 2024 · similarity 0.89)
- Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition (CVPR 2025 · similarity 0.87)
- Semantic-guided Cross-Modal Prompt Learning for Skeleton-based Zero-shot Action Recognition (CVPR 2025 · similarity 0.86)
- Skeleton-based Action Recognition with Non-linear Dependency Modeling and Hilbert-Schmidt Independence Criterion (AAAI 2025 · similarity 0.86)
- Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition (CVPR 2025 · similarity 0.86)
- MaskCLR: Attention-Guided Contrastive Learning for Robust Action Representation Learning (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.