Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Boosting Skeleton-based Zero-Shot Action Recognition with Training-Free Test-Time Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Skeleton-Cache, a training-free test-time adaptation module that adds a non-parametric cache of multi-descriptor skeleton features with LLM-guided descriptor fusion to boost existing skeleton zero-shot action recognition models.
Problem
Skeleton-based zero-shot action recognition models trained only on seen classes suffer distribution shift on unseen actions and cannot adapt at inference; gradient-based TTA is costly and existing training-free TTA is designed for images.
Approach
A non-parametric, class-blocked cache of structured skeleton keys (global, four body-part, and three temporal-segment descriptors) with entropy-based insertion, retrieval per descriptor type, and fusion of descriptor-wise logits using class-specific weights elicited from an LLM (GPT-4o) with no training.
Key results
- PURLS + Skeleton-Cache reaches 85.46% ZSL on NTU60 55/5, up 6.24 points over PURLS (79.22%).
- SA-DVAE + Skeleton-Cache reaches 89.86% ZSL on NTU60 random split and 71.05% on PKU-MMD, up 5.66 and 4.51 points.
- Uniform descriptor weights give 83.67% vs 85.46% with LLM weights on NTU60 55/5, showing LLM priors add gains beyond averaging.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Skeleton-Cache to SA-DVAE | SA-DVAE · NTU RGB+D 60 55/5 | ZSL accuracy: 82.37 -> 89.41 | Helped |
| Add Skeleton-Cache to SA-DVAE | SA-DVAE · NTU RGB+D 60 48/12 | ZSL accuracy: 41.38 -> 47.83 | Helped |
| Add Skeleton-Cache to SA-DVAE | SA-DVAE · NTU RGB+D 120 110/10 | ZSL accuracy: 68.77 -> 74.29 | Helped |
| Add Skeleton-Cache to SA-DVAE | SA-DVAE · NTU RGB+D 120 96/24 | ZSL accuracy: 46.12 -> 53.14 | Helped |
| Use LLM-guided weights instead of random descriptor weights (ablation) | PURLS + Skeleton-Cache with full descriptors · NTU RGB+D splits 55/5, 48/12, 110/10, and 96/24 | ZSL accuracy: 77.31 -> 85.46; 38.56 -> 45.22; 68.02 -> 77.60; 46.47 -> 56.83 | Helped |
| Use LLM-guided weights instead of uniform descriptor weights (ablation) | PURLS + Skeleton-Cache with full descriptors · NTU RGB+D splits 55/5, 48/12, 110/10, and 96/24 | ZSL accuracy: 83.67 -> 85.46; 44.88 -> 45.22; 75.54 -> 77.60; 54.37 -> 56.83 | Helped |
| Use only the global descriptor instead of the full global, spatial, and temporal descriptors (ablation) | PURLS + Skeleton-Cache · NTU RGB+D splits 55/5, 48/12, 110/10, and 96/24 | ZSL accuracy: 82.60 vs 85.46; 42.13 vs 45.22; 73.08 vs 77.60; 53.57 vs 56.83 | Hurt |
| Use only four spatial descriptors instead of the full eight-part descriptor set (ablation) | PURLS + Skeleton-Cache · NTU RGB+D splits 55/5, 48/12, 110/10, and 96/24 | ZSL accuracy: 83.79 vs 85.46; 44.68 vs 45.22; 75.46 vs 77.60; 56.07 vs 56.83 | Hurt |
| Use only three temporal descriptors instead of the full eight-part descriptor set (ablation) | PURLS + Skeleton-Cache · NTU RGB+D splits 55/5, 48/12, 110/10, and 96/24 | ZSL accuracy: 82.14 vs 85.46; 43.32 vs 45.22; 74.52 vs 77.60; 54.41 vs 56.83 | Hurt |
| Use cache updates based on adapted logits instead of zero-shot logits (ablation) | PURLS + Skeleton-Cache · NTU RGB+D 60 55/5 | Top-1 accuracy: 85.46 -> 85.77 | Helped |
| Use cache updates based on adapted logits instead of zero-shot logits (ablation) | PURLS + Skeleton-Cache · NTU RGB+D 60 48/12 | Top-1 accuracy: 45.22 -> 44.66 | Hurt |
| Use cache updates based on adapted logits instead of zero-shot logits (ablation) | PURLS + Skeleton-Cache · NTU RGB+D 120 110/10 | Top-1 accuracy: 77.60 -> 77.84 | Helped |
Surprising findings
- Gradient-based prompt-tuning methods (TPT, DiffTPT) perform far worse than the unadapted PURLS baseline on skeleton zero-shot tasks, attributed to disrupting skeleton-text alignment.
- Random descriptor weights degrade accuracy below the no-weighting baseline on several splits.
Limitations
- Relies on the quality of representations from the frozen SZAR backbone; performance may drop under severe occlusion or pose noise.
- Descriptor weights come from an LLM queried per class, so quality depends on LLM knowledge; paper's prompt results were generated with GPT-4o while appendix mentions gpt-4-turbo.
Experimental setup
Datasets: NTU RGB+D 60 NTU RGB+D 120 PKU-MMD
Benchmarks: NTU RGB+D 60 NTU RGB+D 120 PKU-MMD
Metrics: Top-1 zero-shot accuracy GZSL seen accuracy GZSL unseen accuracy GZSL harmonic mean throughput
Baselines: ReViSE JPoSE CADA-VAE SynSE SMIE PURLS SA-DVAE STAR TPT DiffTPT CALIP AdaNPC
Builds on: Tent TDA CALIP AdaNPC PURLS
Closest papers by meaning
- Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition (CVPR 2025 · similarity 0.89)
- Part-aware Unified Representation of Language and Skeleton for Zero-shot Action Recognition (CVPR 2024 · similarity 0.89)
- Semantic-guided Cross-Modal Prompt Learning for Skeleton-based Zero-shot Action Recognition (CVPR 2025 · similarity 0.89)
- SKI Models: Skeleton Induced Vision-Language Embeddings for Understanding Activities of Daily Living (AAAI 2025 · similarity 0.87)
- Recovering Complete Actions for Cross-dataset Skeleton Action Recognition (NeurIPS 2024 · similarity 0.86)
- MaskCLR: Attention-Guided Contrastive Learning for Robust Action Representation Learning (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.