Research map / Vision-language and multimodal / CLIP and zero-shot
Part-aware Unified Representation of Language and Skeleton for Zero-shot Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PURLS, which aligns GPT-3-generated global, body-part, and temporal-interval action descriptions with skeleton features partitioned adaptively via cross-attention, for zero-shot skeleton action recognition.
Problem
Zero-shot action recognition on skeleton sequences, where global label-level semantic alignment fails to transfer locally shared body-part and temporal movement knowledge from seen to unseen classes.
Approach
Enriches action labels with GPT-3 descriptions of global, body-part-based, and temporal-interval-based movements, and uses a cross-attention adaptive partitioning module to pool skeleton joint features per description, aligning them contrastively with CLIP text embeddings; at test time only the global representation is used.
Key results
- PURLS reaches 79.23% on NTU-RGB+D 60 55/5 split and 40.99% on the 48/12 split, the best among compared methods (SMIE 77.98%, SynSE 75.81%).
- On NTU-RGB+D 120 110/10 split PURLS achieves 71.95% vs 65.74% for SMIE; on the 60/60 split it reaches 19.63%.
- On the newly curated Kinetics-skeleton 200 benchmark, PURLS obtains 32.22% on 180/20 and 11.75% on 120/80, outperforming the Global baseline across splits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add local body-part and temporal alignment with adaptive partitioning to global label alignment | PURLS / Shift-GCN · NTU-RGB+D 60, all four splits | accuracy (%): 64.69 -> 79.23; 35.46 -> 40.99; 27.15 -> 31.05; 16.29 -> 23.52 | Helped |
| Add local body-part and temporal alignment with adaptive partitioning to global label alignment | PURLS / Shift-GCN · NTU-RGB+D 120, all four splits | accuracy (%): 66.96 -> 71.95; 44.27 -> 52.01; 21.31 -> 28.38; 14.12 -> 19.63 | Helped |
| Add local body-part and temporal alignment with adaptive partitioning to global label alignment | PURLS / Shift-GCN · Kinetics-skeleton 200, all four splits | accuracy (%): 25.96 -> 32.22; 15.85 -> 22.56; 10.23 -> 12.01; 7.77 -> 11.75 | Helped |
| Replace Shift-GCN with another tested skeleton encoder (ablation) | PURLS · NTU-RGB+D 60 55/5 | accuracy (%): 79.23; 76.75; 79.97; 80.41; 81.14 | Mixed |
| Replace GPT-3 descriptions with GPT-3.5 descriptions (ablation) | PURLS · NTU-RGB+D 60 | accuracy (%): 79.23 -> 79.17; 40.99 -> 40.98; 31.05 -> 30.07; 23.52 -> 19.95 | Mixed |
| Replace GPT-3 descriptions with GPT-4 descriptions (ablation) | PURLS · NTU-RGB+D 60 | accuracy (%): 79.23 -> 81.53; 40.99 -> 41.90; 31.05 -> 27.28; 23.52 -> 21.45 | Mixed |
| Use GPT-3 action descriptions rather than original label semantics for global alignment only (ablation) | Global · NTU-RGB+D 60 | accuracy (%): 64.69 -> 78.50; 35.46 -> 33.47; 27.15 -> 29.21; 16.29 -> 22.27 | Mixed |
| Replace adaptive partitioning with static partitioning (ablation) | PURLS · NTU-RGB+D 60 and NTU-RGB+D 120 | accuracy (%): NTU60: 76.46, 33.03, 29.57, 22.00; NTU120: 67.62, 46.83, 26.98, 18.03 | Mixed |
| Add body-part-based alignment (ablation) | Global description alignment with PURLS partitioning · NTU-RGB+D 60 and NTU-RGB+D 120 | accuracy (%): Average: 76.68, 37.80, 30.92, 22.20; 68.11, 30.93, 24.36, 18.67. Learnable: 76.32, 37.62, 29.06, 21.91; 71.73, 40.92, 23 | Mixed |
| Add temporal-interval-based alignment (ablation) | Global description alignment with PURLS partitioning · NTU-RGB+D 60 and NTU-RGB+D 120 | accuracy (%): Average: 78.65, 38.80, 28.14, 22.69; 55.73, 50.67, 27.50, 17.50. Learnable: 77.70, 40.69, 28.84, 22.46; 71.26, 46.13, 24 | Mixed |
| Add both body-part and temporal-interval alignment (ablation) | Global description alignment with PURLS partitioning · NTU-RGB+D 60 and NTU-RGB+D 120 | accuracy (%): Average: 79.02, 39.92, 31.00, 23.47; 73.55, 51.38, 27.67, 18.66. Learnable: 79.23, 40.99, 31.05, 23.52; 71.95, 52.01, 28 | Mixed |
| Use learnable rather than average alignment-loss weights (ablation) | PURLS with local alignment · NTU-RGB+D 60 and NTU-RGB+D 120 | accuracy (%): Average and learnable results are reported in Table 6 for body-part-only, temporal-only, and combined alignment. | Mixed |
Surprising findings
- Using GPT-3 descriptions for global alignment alone does not consistently improve over label-level alignment on NTU-RGB+D 60 (e.g. 78.50 vs 64.69 on 55/5 but lower on 120 110/10 for Global-GPT3 vs Global-original is higher; mixed results overall).
- Static body-part partitioning gives unstable gains, while adaptive partitioning yields consistent improvement.
Limitations
- Kinetics-skeleton 200 accuracy drops sharply as unseen classes increase, attributed to limits of pre-trained visual backbones.
- Relies on a single fixed GPT-3 question and prompt template for description generation; multiple descriptions not explored in depth.
- Gains from GPT-3/GPT-4 description replacement vary across splits (e.g. GPT-4 PURLS lower than GPT-3 on 40/20).
Experimental setup
Datasets: NTU-RGB+D 60 NTU-RGB+D 120 Kinetics-skeleton 200
Benchmarks: NTU-RGB+D 60 NTU-RGB+D 120 Kinetics-skeleton 200
Metrics: zero-shot accuracy (%)
Baselines: DeViSE ReViSE JPoSE CADA-VAE SynSE SMIE Global baseline
Builds on: CLIP Shift-GCN SynSE DeViSE
Closest papers by meaning
- Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition (CVPR 2025 · similarity 0.91)
- Semantic-guided Cross-Modal Prompt Learning for Skeleton-based Zero-shot Action Recognition (CVPR 2025 · similarity 0.90)
- Boosting Skeleton-based Zero-Shot Action Recognition with Training-Free Test-Time Adaptation (NeurIPS 2025 · similarity 0.89)
- Heterogeneous Skeleton-Based Action Representation Learning (CVPR 2025 · similarity 0.89)
- Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition (CVPR 2025 · similarity 0.88)
- Universal Skeleton Understanding via Differentiable Rendering and MLLMs (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.