Research map / Vision-language and multimodal / CLIP and zero-shot
Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Neuron, a zero-shot skeleton action recognition framework that uses multi-turn LLM-generated context-aware side information and evolving spatial-temporal micro-prototypes (spatial compression and temporal memory) to align skeleton and semantic features from micro to macro levels.
Problem
Zero-shot skeleton action recognition relies on homogeneous, one-shot semantic descriptions and uncontrolled global alignment, which fails to capture fine-grained skeleton-semantic correspondence for unseen action generalization.
Approach
Generate multi-phase, multi-granularity LLM descriptions via multi-turn prompting and use learnable spatial and temporal micro-prototypes that progressively grow via a spatial compression mechanism (top-alpha attribute selection) and a temporal memory mechanism (recall/remember gating), aligned stepwise to phase-specific semantics.
Key results
- On NTU RGB+D 60 Xsub 55/5 split, ZSL accuracy 86.9% vs. STAR 81.4%.
- On NTU RGB+D 60 Xsub 48/12 GZSL, harmonic mean 59.1% vs. STAR 46.6% (Xview 61.5 H with 48/12 split reported as 59.1 for Xsub).
- On PKU-MMD I Xsub 46/5 split, ZSL accuracy 89.2% vs. STAR 76.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Neuron instead of SA-DAVE | Zero-shot skeleton action recognition · NTU RGB+D 60 Xsub 55/5 | ZSL accuracy: 82.4 -> 86.9 | Helped |
| Use Neuron instead of STAR | Zero-shot skeleton action recognition · NTU RGB+D 60 Xsub 48/12 | ZSL accuracy: 45.1 -> 62.7 | Helped |
| Use Neuron instead of STAR | Zero-shot skeleton action recognition · PKU-MMD I Xsub 46/5 | ZSL accuracy: 76.3 -> 89.2 | Helped |
| Add spatial granularity updating (ablation) | Neuron without components · NTU RGB+D 60 Xsub 55/5 | ZSL accuracy: 79.9 -> 81.0 | Helped |
| Add spatial compression (ablation) | Neuron with spatial granularity updating · NTU RGB+D 60 Xsub 55/5 | ZSL accuracy: 81.0 -> 81.2 | Helped |
| Add phase evolving and spatial compression (ablation) | Neuron with spatial granularity updating · NTU RGB+D 60 Xsub 55/5 | ZSL accuracy: 81.2 -> 81.7 | Helped |
| Add temporal memory (ablation) | Neuron with spatial and temporal evolution and spatial compression · NTU RGB+D 60 Xsub 55/5 | ZSL accuracy: 81.7 -> 86.9 | Helped |
| Add spatial compression (ablation) | Neuron without spatial compression · NTU RGB+D 60 Xsub 55/5 | GZSL harmonic mean: 67.1 -> 69.9 | Helped |
Surprising findings
- Removing the temporal memory mechanism drops performance by around 5.2% ZSL, showing memory is key to avoiding knowledge oblivion across phases.
Limitations
- Fails on extremely similar and abstract actions (e.g., nausea/vomiting vs sneeze/cough) where LLM side information remains homogeneous.
- Relies on LLM-generated descriptions whose quality and homogeneity remain a bottleneck.
Experimental setup
Datasets: NTU RGB+D 60 NTU RGB+D 120 PKU-MMD I
Benchmarks: NTU RGB+D 60 NTU RGB+D 120 PKU-MMD I
Metrics: Top-1 accuracy seen accuracy unseen accuracy harmonic mean
Baselines: ReViSE JPoSE CADA-VAE SynSE SMIE PURLS SA-DAVE STAR
Builds on: CLIP STAR Shift-GCN calibrated stacking
Closest papers by meaning
- Part-aware Unified Representation of Language and Skeleton for Zero-shot Action Recognition (CVPR 2024 · similarity 0.91)
- Heterogeneous Skeleton-Based Action Representation Learning (CVPR 2025 · similarity 0.90)
- Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition (CVPR 2025 · similarity 0.90)
- Semantic-guided Cross-Modal Prompt Learning for Skeleton-based Zero-shot Action Recognition (CVPR 2025 · similarity 0.90)
- Boosting Skeleton-based Zero-Shot Action Recognition with Training-Free Test-Time Adaptation (NeurIPS 2025 · similarity 0.89)
- Skeleton-based Action Recognition with Non-linear Dependency Modeling and Hilbert-Schmidt Independence Criterion (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.