Research map / Video / Action recognition
Recovering Complete Actions for Cross-dataset Skeleton Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a recover-and-resample temporal augmentation for single-domain generalization in cross-dataset skeleton action recognition, using clustered boundary poses and smooth linear transforms learned from a complete action prior.
Problem
Skeleton-based action recognition degrades sharply when transferred across datasets, largely due to temporal mismatch (partial vs. complete action segments) between datasets; the paper targets single-domain generalization for this setting.
Approach
Exploit a complete action prior: recover complete actions from partial training sequences via two steps (boundary-pose-conditioned linear-interpolation extrapolation, then clustered context-aware smooth linear temporal transforms), then random-crop resample them as strong augmentations learned with k-means.
Key results
- Average accuracy over four cross-dataset sub-settings improves from 54.4 (ERM) to 60.1, outperforming the second-best ST-Cubism (57.1).
- With CTR-GCN backbone, average accuracy rises from 56.3 to 64.3; with ST-GCN, from 50.4 to 55.4.
- On N51->P51 with HCN, accuracy is 62.8 vs. 61.3 for ST-Cubism.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply recover-and-resample augmentation | AGCN · Cross-dataset average (four settings) | balanced accuracy: 54.4 -> 60.1 | Helped |
| apply recover-and-resample augmentation | AGCN · N51→P51 | accuracy: 70.5 -> 72.2 | Helped |
| apply recover-and-resample augmentation | HCN · P51→N51 | accuracy: 53.8 -> 53.3 | Hurt |
| apply linear transforms only (ablation) | ERM · Cross-dataset average (four settings) | balanced accuracy: 54.4 -> 57.2 | Helped |
| apply boundary-conditioned extrapolation only (ablation) | ERM · Cross-dataset average (four settings) | balanced accuracy: 54.4 -> 57.8 | Helped |
| learn boundary poses and linear transforms from P̄ instead of source data (ablation) | Recover-and-resample · Cross-dataset average (four settings) | balanced accuracy: 60.1 -> 60.5 | Helped |
| reduce the number of linear transform clusters to 3 (ablation) | Recover-and-resample · Cross-dataset average (four settings) | balanced accuracy: 60.1 -> 58.9 | Hurt |
| remove boundary pose clustering and extend from the raw first frame (ablation) | Recover-and-resample · Cross-dataset average (four settings) | balanced accuracy: 60.1 -> 58.8 | Hurt |
| replace interpolation with neural extrapolation (ablation) | Recover-and-resample · Cross-dataset average (four settings) | balanced accuracy: 60.1 -> 58.8 | Hurt |
| replace linear interpolation with neural infilling (ablation) | Recover-and-resample · Cross-dataset average (four settings) | balanced accuracy: 60.1 -> 58.7 | Hurt |
| use latent-space rather than joint-space representations (ablation) | Recover-and-resample · Cross-dataset average (four settings) | balanced accuracy: 60.1 -> 59.8 | Hurt |
| replace k-means with agglomerative clustering (ablation) | Recover-and-resample · Cross-dataset average (four settings) | balanced accuracy: 60.1 -> 59.5 | Hurt |
Surprising findings
- Directly extending with the raw first frame (no boundary pose clustering) hurts performance by about 1.3%.
- Learning-based neural extrapolation or infilling networks underperform the parameter-free linear interpolation design.
Limitations
- Evaluated only on 18-class indoor shared subsets of three datasets plus limited Kinetics 12-class setting.
- Resampling stage is fixed to CropResize and not optimized; authors note the approach may need better temporal encoding of resampling information.
- Gains on the P51->N51 direction are smaller than the full ST-Cubism method.
Experimental setup
Datasets: NTU60-RGBD PKU-MMD ETRI-Activity3D Kinetics
Benchmarks: NTU->ETRI (N->E) NTU->PKU (N->P) ETRI Adult->NTU (E_A->N) ETRI Adult->PKU (E_A->P) N51->P51 P51->N51 N12->K12
Metrics: balanced accuracy average accuracy
Baselines: ERM CCSA ADA ST-Cubism Skeleton-MAE HICLR Mixup CropPad CropResize Uniform sampling TSN OTAM+kNN
Builds on: ST-Cubism CropResize AGCN OTAM
Closest papers by meaning
- Stitch, Contrast, and Segment: Learning a Human Action Segmentation Model Using Trimmed Skeleton Videos (AAAI 2025 · similarity 0.89)
- Navigating Open Set Scenarios for Skeleton-Based Action Recognition (AAAI 2024 · similarity 0.89)
- Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition (CVPR 2025 · similarity 0.88)
- Heterogeneous Skeleton-Based Action Representation Learning (CVPR 2025 · similarity 0.88)
- Skeleton-based Action Recognition with Non-linear Dependency Modeling and Hilbert-Schmidt Independence Criterion (AAAI 2025 · similarity 0.88)
- Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.