Research map / Video / Action recognition
VA-AR: Learning Velocity-Aware Action Representations with Mixture of Window Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes how skeleton-based action recognition degrades with action velocity and proposes VA-AR, a Transformer with Mixture of Window Attention that adapts temporal window size to action speed.
Problem
Skeleton-based action recognition methods degrade as action velocity increases, since fixed temporal receptive fields do not suit both slow daily actions and fast athletic actions.
Approach
Multi-scale local and shifted window attention over the temporal axis, combined with a token-level mixture-of-experts router that softly weights window-size experts, so the model adaptively picks the temporal receptive field per sample and frame.
Key results
- Ensemble of four modalities reaches 93.1% on NTU-60 X-Sub, 90.3% on NTU-120 X-Sub, 73.9% on P2A, 73.4% on Olympic Badminton and 92.8% on FineGym.
- Joint-only VA-AR (GCN+VA-Trans.) reaches 90.1 on NTU-60 X-Sub and 73.5 on P2A, with reported gains of 1.23%, 1.52% and 2.54% on P2A, Olympic Badminton and FineGym.
- Ablation on P2A: LWA+SWA adds about 18.7 points over plain Transformer (58.4 to 69.3), multi-scale adds 1.9, and MoE adds 2.3 to reach 73.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LWA and SWA to GCN+Trans. (ablation) | GCN+Trans. · P2A (joint modality) | accuracy: 58.4 -> 69.3 | Helped |
| Add multi-scale windows to LWA and SWA. (ablation) | GCN+Trans. · P2A (joint modality) | accuracy: 69.3 -> 71.2 | Helped |
| Add MoE to multi-scale LWA and SWA. (ablation) | GCN+Trans. · P2A (joint modality) | accuracy: 71.2 -> 73.5 | Helped |
| Increase experts per window from 1 to 2 with a single size-4 window. (ablation) | VA-Transformer · P2A | accuracy: 68.1 -> 70.3 | Helped |
| Increase experts per window from 1 to 2 with a single size-4 window. (ablation) | VA-Transformer · Olympic Badminton | accuracy: 67.5 -> 69.8 | Helped |
| Increase experts per window from 1 to 2 with a single size-8 window. (ablation) | VA-Transformer · P2A | accuracy: 69.3 -> 71.5 | Helped |
| Increase experts per window from 1 to 2 with a single size-16 window. (ablation) | VA-Transformer · Olympic Badminton | accuracy: 69.0 -> 70.7 | Helped |
| Add a third window size, 16, to windows {4, 8}. (ablation) | VA-Transformer · P2A | accuracy: 71.9 -> 73.5 | Helped |
| Add a third window size, 16, to windows {4, 8}. (ablation) | VA-Transformer · Olympic Badminton | accuracy: 71.7 -> 73.2 | Helped |
| Add window size 32 to windows {4, 8, 16}. (ablation) | VA-Transformer · P2A | accuracy: 73.5 -> 73.3 | Hurt |
| Add window size 32 to windows {4, 8, 16}. (ablation) | VA-Transformer · Olympic Badminton | accuracy: 73.2 -> 73.1 | Hurt |
| Use VA-AR instead of the strongest listed baseline. | Skeleton action recognition · NTU-60 X-View | accuracy: 97.5 -> 97.2 | Hurt |
Surprising findings
- Adding motion modalities gives smaller gains on sports datasets than on daily-action datasets, which the authors attribute to small windows already capturing frame-difference motion.
Limitations
- Velocity is defined from a three-frame joint calculation and speed-binned evaluation depends on this proxy.
- Gains over baselines on NTU-60 and NTU-120 are small, and the Transformer-only variant trails several GCN baselines.
Experimental setup
Datasets: NTU RGB+D 60 NTU RGB+D 120 P2A Olympic Badminton FineGym
Benchmarks: NTU RGB+D 60 NTU RGB+D 120 P2A Olympic Badminton FineGym
Metrics: top-1 accuracy
Baselines: ST-GCN AS-GCN 2S-AGCN MS-G3D CTR-GCN ST-TR ASE-GCN InfoGCN ZoomTrans HD-GCN BlockGCN DS-GCN
Builds on: Swin Transformer mixture of attention ST-GCN CTR-GCN
Closest papers by meaning
- AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions (CVPR 2018 · similarity 0.89)
- Memory-Augmented Temporal Dynamic Learning for Action Recognition (AAAI 2019 · similarity 0.87)
- Attentional Pooling for Action Recognition (NeurIPS 2017 · similarity 0.87)
- Action Detail Matters: Refining Video Recognition with Local Action Queries (CVPR 2025 · similarity 0.86)
- SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization (AAAI 2025 · similarity 0.85)
- TIM: A Time Interval Machine for Audio-Visual Action Recognition (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.