Research map / Vision-language and multimodal / CLIP and zero-shot

Semantic-guided Cross-Modal Prompt Learning for Skeleton-based Zero-shot Action Recognition

CVPR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Attribute-based zero-shot learning · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SCoPLe, a data-driven cross-modal prompt learning framework (dual-stream language prompting plus semantic-guided joint-shaped skeleton prompting) for zero-shot skeleton-based action recognition, evaluated on NTU-60, NTU-120 and PKU-MMD.

Problem

Zero-shot skeleton-based action recognition needs effective knowledge transfer from seen to unseen action classes, but existing methods depend on manually crafted action descriptions or assumptions, and naive CLIP prompt learning gives unstable gains and hurts generalization.

Approach

Replaces manual semantic guidance with learnable prompts on both branches: a dual-stream language prompt that blends the frozen CLIP attention output with a prompted stream via learnable weights, and joint-shaped visual prompts whose features are pooled by an adaptive sampler that uses the text embedding as query.

Key results

• On NTU-RGB+D 60 55/5 split, SCoPLe reaches 84.10% ZSL accuracy versus 82.37% for SA-DVAE (prior SOTA).
• On NTU-RGB+D 120 96/24 split, SCoPLe reports 52.17% ZSL accuracy, and 52.23 GZSL H versus 44.50 for SA-DVAE.
• On random 3-split PKU-MMD (46/5), SCoPLe achieves 71.41% ZSL accuracy, a reported +4.87% gain over SA-DVAE.

Measured effects reported in the paper

Surprising findings

• Applying CoOp or MaPLe prompting to CLIP for ZSSAR yields volatile gains, sometimes lower than the baseline, on NTU-60 48/12 and NTU-120 splits.
• Skeleton prompting without language-side guidance gives unstable improvement, indicating ZSL depends strongly on text embedding quality.

Limitations

• Less interpretable than manually refined description-based methods such as PURLS.
• Results on some splits (e.g., 48/12 NTU-60 GZSL and PKU-MMD GZSL) are not uniformly best; gains are split-dependent.
• Generalized ZSSAR relies on an entropy-threshold gating heuristic with a pre-defined threshold.

Experimental setup

Datasets: NTU-RGB+D 60 NTU-RGB+D 120 PKU-MMD

Benchmarks: NTU-RGB+D 60 NTU-RGB+D 120 PKU-MMD

Metrics: zero-shot accuracy generalized zero-shot harmonic mean (H) seen accuracy unseen accuracy

Baselines: ReViSE JPoSE CADA-VAE SynSE SMIE STAR PURLS SA-DVAE CLIP-CoOp CLIP-MaPLe

Builds on: CoOp MaPLe CLIP SynSE SA-DVAE

Closest papers by meaning

• Part-aware Unified Representation of Language and Skeleton for Zero-shot Action Recognition (CVPR 2024 · similarity 0.90)
• Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition (CVPR 2025 · similarity 0.90)
• Boosting Skeleton-based Zero-Shot Action Recognition with Training-Free Test-Time Adaptation (NeurIPS 2025 · similarity 0.89)
• Heterogeneous Skeleton-Based Action Representation Learning (CVPR 2025 · similarity 0.87)
• Navigating Open Set Scenarios for Skeleton-Based Action Recognition (AAAI 2024 · similarity 0.86)
• Recovering Complete Actions for Cross-dataset Skeleton Action Recognition (NeurIPS 2024 · similarity 0.86)