Research map / Video / Action recognition
Spatio-Temporal Fusion for Human Action Recognition via Joint Trajectory Graph
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JT-GraphFormer, which builds a joint trajectory graph over multi-frame skeleton groups, applies Dijkstra-distance-weighted attention (STDA), and classifies with a Koopman operator for skeleton-based action recognition.
Problem
Skeleton-based human action recognition struggles to capture joint spatio-temporal dependencies: GCNs use only spatial topology and Transformer self-attention lacks topological structure information.
Approach
A Joint Trajectory Graph connects joints across neighboring frames into one uniform graph; a Spatio-Temporal Dijkstra Attention multiplies attention scores by exp(-shortest-path distance) plus a learnable bias; and a Koopman operator fitted via DMD serves as the classifier.
Key results
- JT-GraphFormer (4s) reaches 93.4% X-Sub and 97.5% X-View on NTU RGB+D, above InfoGCN (6s) at 93.0% and 97.1%.
- Joint-only JT-GraphFormer reaches 91.8% X-Sub on NTU RGB+D, and the 4-stream ensemble reaches 89.9% X-Sub on NTU RGB+D 120.
- On N-UCLA, the 4-stream model reaches 97.2%, a 0.4% gain over CTR-GCN+FR (4s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse JTG frame counts 2, 4, 6, and 8 rather than using N=1 (ablation) | JT-GraphFormer · NTU-60 X-Sub | Top-1 accuracy (%): 87.4 -> 91.7 | Helped |
| Use JTG frame count N=6 rather than N=1 (ablation) | JT-GraphFormer · NTU-60 X-Sub | Top-1 accuracy (%): 87.4 -> 90.4 | Helped |
| Use a dynamic, shared JTG without normalization (ablation) | JT-GraphFormer · NTU-60 X-Sub | Top-1 accuracy (%): 89.8 -> 90.4 | Helped |
| Use a dynamic, normalized, shared JTG (ablation) | JT-GraphFormer · NTU-60 X-Sub | Top-1 accuracy (%): 89.8 -> 89.8 | No clear effect |
| Replace the fully connected classifier with a temporal Koopman operator (ablation) | JT-GraphFormer with joint modality · NTU-60 X-Sub | Top-1 accuracy (%): 90.4 -> 90.5 | Helped |
| Use a spatial Koopman operator for motion modalities (ablation) | JT-GraphFormer · NTU-60 X-Sub, joint motion and bone motion | Top-1 accuracy (%): 87.9 -> 88.4; 86.9 -> 87.3 | Helped |
| Ensemble joint and bone modalities rather than using joint only (ablation) | JT-GraphFormer · NTU-60 X-Sub | Top-1 accuracy (%): 91.8 -> 92.5 | Helped |
| Ensemble four streams rather than using joint only (ablation) | JT-GraphFormer · NTU-60 X-Sub | Top-1 accuracy (%): 91.8 -> 93.4 | Helped |
Surprising findings
- Spatial Koopman linearization outperformed temporal linearization on motion modalities, which the authors attribute to the abstraction of temporal evolution.
Limitations
- Koopman operator adds large parameter count (14.54M on NTU-120), making Kinetics-400 testing challenging.
- Only four streams are fused and more stream combinations are unexplored.
- Limited to structured skeleton sequences and not evaluated in unsupervised settings.
Experimental setup
Datasets: NTU RGB+D NTU RGB+D 120 N-UCLA
Benchmarks: NTU RGB+D NTU RGB+D 120 N-UCLA
Metrics: Top-1 accuracy
Baselines: ST-GCN 2s-AGCN DGNN Dynamic-GCN SGN DDGCN DC-GCN+ADG MS-G3D MST-GCN CTR-GCN InfoGCN STF
Builds on: Graphormer ST-GCN CTR-GCN 2s-AGCN dynamic mode decomposition
Closest papers by meaning
- Spatio-Temporal Graph Routing for Skeleton-Based Action Recognition (AAAI 2019 · similarity 0.93)
- Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition (AAAI 2021 · similarity 0.91)
- Dynamic Semantic-Based Spatial Graph Convolution Network for Skeleton-Based Human Action Recognition (AAAI 2024 · similarity 0.89)
- Are Spatial-Temporal Graph Convolution Networks for Human Action Recognition Over-Parameterized? (CVPR 2025 · similarity 0.89)
- BlockGCN: Redefine Topology Awareness for Skeleton-Based Action Recognition (CVPR 2024 · similarity 0.89)
- Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.