Research map / Architectures and efficiency / Attention and transformers
Behavioral Recognition of Skeletal Data Based on Targeted Dual Fusion Strategy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FRF-GCN, a lightweight skeleton-based action recognition model that combines front fusion (joint+bone, joint motion+bone motion) with rear score fusion, targeted adjacency matrices, and a parallel spatial-temporal-channel attention module.
Problem
Multi-stream skeleton-based behavioral recognition achieves high accuracy but with high model complexity and parameter counts, and fixed adjacency matrices and serially-applied attention limit performance on diverse actions.
Approach
Fuse streams in pairs before the network (front fusion) and combine the two resulting streams by score fusion (rear fusion), with a different adjacency matrix per fused stream (AJB for joint+bone, AJBM for joint-motion+bone-motion), plus an STC-P module that computes spatial, temporal and channel attention in parallel, and multi-field depth-point temporal convolution (MD-TGCN) to cut parameters.
Key results
- FRF-GCN reaches 91.3% cross-subject and 96.5% cross-view accuracy on NTU RGB+D 60 with 2.42M parameters.
- On NTU RGB+D 120, FRF-GCN gets 87.1% cross-subject and 88.4% cross-setup accuracy, below 4s-ACE-Ens and ML-STGNet on that dataset.
- On Kinetics-Skeleton 400, FRF-GCN reaches 37.9% top-1 and 60.7% top-5, below MS-G3D's 38.0% top-1 and ML-STGNet's 38.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the targeted AJB/AJBM adjacency matrices with I + Aske for both fusion streams (ablation) | FRF-GCN · NTU RGB+D 60 Cross-Subject | accuracy: 91.29 -> 90.19 | Hurt |
| use I for joint+bone and I + Aske for joint-motion+bone-motion (ablation) | FRF-GCN · NTU RGB+D 60 Cross-Subject | accuracy: 91.29 -> 89.57 | Hurt |
| use I + Aske for joint+bone and I for joint-motion+bone-motion (ablation) | FRF-GCN · NTU RGB+D 60 Cross-Subject | accuracy: 91.29 -> 90.03 | Hurt |
| use I + Aske for joint+bone and AJB for joint-motion+bone-motion (ablation) | FRF-GCN · NTU RGB+D 60 Cross-Subject | accuracy: 91.29 -> 90.45 | Hurt |
| use AJB for joint+bone and I + Aske for joint-motion+bone-motion (ablation) | FRF-GCN · NTU RGB+D 60 Cross-Subject | accuracy: 91.29 -> 91.19 | Hurt |
| use AJB for both fusion streams (ablation) | FRF-GCN · NTU RGB+D 60 Cross-Subject | accuracy: 91.29 -> 91.18 | Hurt |
| use AJBM for joint+bone and AJB for joint-motion+bone-motion (ablation) | FRF-GCN · NTU RGB+D 60 Cross-Subject | accuracy: 91.29 -> 91.10 | Hurt |
| use AJBM for both fusion streams (ablation) | FRF-GCN · NTU RGB+D 60 Cross-Subject | accuracy: 91.29 -> 90.90 | Hurt |
| remove the temporal-spatial feature fusion step in STC-P attention (ablation) | FRF-GCN · NTU RGB+D 60 Cross-Subject | accuracy: 91.29 -> 90.95 | Hurt |
| fuse temporal, spatial, and channel attention before extracting their attention (ablation) | FRF-GCN · NTU RGB+D 60 Cross-Subject | accuracy: 91.29 -> 91.18 | Hurt |
| replace multi-field depth-point temporal convolution with conventional TCN (ablation) | FRF-GCN · NTU RGB+D 60 Cross-Subject | accuracy: 91.29 -> 90.13 | Hurt |
| use pure rear fusion instead of front-rear dual fusion (ablation) | FRF-GCN · NTU RGB+D 60 Cross-Subject | accuracy: 91.29 -> 91.56 | Helped |
Surprising findings
- Using the more-connected adjacency matrix AJBM for all streams performs worse than using AJB for all streams, suggesting more joint relations do not necessarily help recognition.
Limitations
- Does not fully discriminate extremely similar actions, per the authors.
- Slightly below some SOTA methods on NTU RGB+D 120 and Kinetics-Skeleton 400.
- Ablations were run only on NTU60 cross-subject.
Experimental setup
Datasets: NTU RGB+D 60 NTU RGB+D 120 Kinetics-Skeleton 400
Benchmarks: NTU RGB+D 60 NTU RGB+D 120 Kinetics-Skeleton 400
Metrics: top-1 accuracy parameter count (M)
Baselines: ST-GCN 2S-AGCN MS-G3D MS-AAGCN EfficientGCN-B0 MST-GCN 4s-ACE-Ens ML-STGNet 2M-STGCN STF-Net
Builds on: ST-GCN 2S-AGCN MS-AAGCN EfficientGCN CTR-GCN
Closest papers by meaning
- Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition (CVPR 2019 · similarity 0.88)
- An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition (CVPR 2019 · similarity 0.87)
- BlockGCN: Redefine Topology Awareness for Skeleton-Based Action Recognition (CVPR 2024 · similarity 0.86)
- Dynamic Semantic-Based Spatial Graph Convolution Network for Skeleton-Based Human Action Recognition (AAAI 2024 · similarity 0.85)
- Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition (CVPR 2025 · similarity 0.85)
- Are Spatial-Temporal Graph Convolution Networks for Human Action Recognition Over-Parameterized? (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.