Research map / Video / Action recognition
Stitch, Contrast, and Segment: Learning a Human Action Segmentation Model Using Trimmed Skeleton Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Stitch, Contrast, and Segment (SCS), which stitches trimmed skeleton videos into multi-action sequences, pretrains an ST-GCN encoder with granular contrastive learning, and adds a TCN segmentation layer for skeleton action segmentation under supervised and zero-shot temporal domain adaptation.
Problem
Skeleton-based action segmentation normally requires costly frame-wise annotated untrimmed skeleton videos; the paper asks whether models trained only on trimmed skeleton videos can segment untrimmed videos.
Approach
Treat trimmed skeleton clips as elementary motions and stitch them via frame correspondence (spatial joint alignment) into plausible multi-action sequences; use these stitched sequences for granular contrastive pretraining (InfoNCE over class-level pooled encodings from different permutations) of an ST-GCN encoder, then attach a TCN segmentation head, avoiding sliding windows.
Key results
- Supervised adaptation NTU to PKU-MMD v2 (E2E) reaches 60.9 mIoU and 81.1 Acc, outperforming Hi-TRS (55.0 Acc) and other sliding-window baselines.
- Zero-shot adaptation NTU to PKU-MMD v1 (E2E) reaches 53.6 mIoU and 61.3 Acc, versus 10.5 mIoU and 18.9 Acc for the baseline.
- PKU1 to PKU2 supervised E2E achieves 71.4 mIoU and 84.6 Acc, versus 51.8 mIoU and 68.1 Acc for the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SCS supervised adaptation instead of LAC | Skeleton action segmentation · NTU to PKU-MMD v1 | mAP@0.1: 61.8 -> 80.9 | Helped |
| Use SCS supervised adaptation instead of LAC | Skeleton action segmentation · NTU to PKU-MMD v1 | mAP@0.5: 90.6 -> 91.1 | Helped |
| Use SCS supervised adaptation instead of ActCLR | Skeleton action segmentation · NTU to PKU-MMD v2 | mIoU: 21.4 -> 60.9 | Helped |
| Use SCS supervised adaptation instead of Hi-TRS | Skeleton action segmentation · NTU to PKU-MMD v2 | accuracy: 55.0 -> 81.1 | Helped |
| Use stitched sequences for zero-shot adaptation instead of the trimmed-data baseline | Skeleton action segmentation · NTU to PKU-MMD v1 | mIoU: 10.5 -> 53.6 | Helped |
| Use stitched sequences for zero-shot adaptation instead of the trimmed-data baseline | Skeleton action segmentation · NTU to PKU-MMD v2 | mIoU: 5.4 -> 20.8 | Helped |
| Register frame correspondence when stitching rather than concatenate randomly (ablation) | End-to-end supervised segmentation · PKU-MMD v1, PKU-MMD v2, and Toyota Smarthome Untrimmed | accuracy | Helped |
| Randomly concatenate trimmed sequences without frame correspondence (ablation) | End-to-end supervised segmentation · PKU-MMD v1, PKU-MMD v2, and Toyota Smarthome Untrimmed | accuracy | Hurt |
| Augment the original untrimmed dataset with frame-correspondence stitched data (ablation) | End-to-end supervised segmentation · PKU-MMD v1, PKU-MMD v2, and Toyota Smarthome Untrimmed | accuracy | Helped |
| Use stitched-data GCL instead of classical spatial and temporal augmentation alone (ablation) | Contrastive pre-training for action segmentation · PKU-MMD v1, PKU-MMD v2, and Toyota Smarthome Untrimmed | accuracy | Helped |
| Combine classical augmentation with Skeleton-Stitch (ablation) | Contrastive pre-training for action segmentation · PKU-MMD v1, PKU-MMD v2, and Toyota Smarthome Untrimmed | accuracy | Helped |
| Train with stitched sequences to expose the model to more action contexts (ablation) | Skeleton action segmentation · PKU-MMD v2 Cross-Sequence protocol | accuracy | Helped |
Surprising findings
- Training on randomly stitched sequences without frame correspondence (Dataset I) leads to drastic performance drop, showing that stitching plausibility matters.
- Classical augmentations (spatial transform, temporal scaling) give limited transferability for segmentation pretraining, though they are complementary to stitching.
Limitations
- Zero-shot results on PKU-MMD v2 are modest (about 21 mIoU), indicating a gap versus supervised adaptation.
- Stitched sequences rely on frame-correspondence thresholds and a search range hyperparameter (beta), requiring tuning.
- Evaluation covers a limited set of trimmed-untrimmed dataset pairs.
Experimental setup
Datasets: NTU RGB+D PKU-MMD v1 PKU-MMD v2 Toyota Smarthome Toyota Smarthome Untrimmed
Benchmarks: PKU-MMD v1 PKU-MMD v2 Toyota Smarthome Untrimmed
Metrics: mAP@0.1 mAP@0.5 mIoU Acc mAPf
Baselines: LAC CrosSCLR Hi-TRS AimCLR ActCLR MS-GCN Baseline E2E
Builds on: ST-GCN MS-GCN InfoNCE MoCo-style memory bank
Closest papers by meaning
- Frame-Level Label Refinement for Skeleton-Based Weakly-Supervised Action Recognition (AAAI 2023 · similarity 0.90)
- Recovering Complete Actions for Cross-dataset Skeleton Action Recognition (NeurIPS 2024 · similarity 0.89)
- Heterogeneous Skeleton-Based Action Representation Learning (CVPR 2025 · similarity 0.89)
- Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition (CVPR 2025 · similarity 0.88)
- MaskCLR: Attention-Guided Contrastive Learning for Robust Action Representation Learning (CVPR 2024 · similarity 0.87)
- Skeleton-based Action Recognition with Non-linear Dependency Modeling and Hilbert-Schmidt Independence Criterion (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.