Research map / Video / Action recognition
Just Add ?! Pose Induced Video Transformers for Understanding Activities of Daily Living
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes π-ViT, a video transformer (TimeSformer) that induces 2D and 3D skeleton knowledge into RGB token representations via auxiliary tasks during training only, so poses are not needed at inference.
Problem
Video transformers rely solely on RGB, which struggles on Activities of Daily Living (ADL) with visually similar fine-grained actions and multi-view capture; existing RGB+pose fusion needs depth sensors or costly pose estimation at inference.
Approach
Two plug-in modules, 2D-SIM (token-skeleton joint presence classification via BCE) and 3D-SIM (MSE feature alignment to a frozen 3D skeleton model plus skeleton-feature classification), are attached to a video transformer during training and removed at inference, inducing pose knowledge into RGB representations.
Key results
- On Toyota-Smarthome CS, π-ViT (RGB only at inference) reaches 72.9% mean class accuracy vs 68.4% for the TimeSformer baseline.
- On NTU120 CS, π-ViT (RGB only) reaches 91.9% top-1 vs 90.6% for TimeSformer.
- On NTU60 CS, π-ViT (RGB only) reaches 94.0% top-1 vs 93.0% for TimeSformer, state-of-the-art among RGB-only methods on that protocol.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 2D-SIM and 3D-SIM to TimeSformer | TimeSformer · Toyota-Smarthome CS, CV1, and CV2 | mCA: 68.4 -> 72.9; 50.0 -> 55.2; 60.6 -> 64.8 | Helped |
| add 2D-SIM and 3D-SIM | TimeSformer · NTU120 CS and CSet | top-1 accuracy: 90.6 -> 91.9; 91.6 -> 92.9 | Helped |
| add 2D-SIM and 3D-SIM | TimeSformer · NTU60 CS and CV | top-1 accuracy: 93.0 -> 94.0; 97.2 -> 97.9 | Mixed |
| add 2D-SIM instead of using the baseline alone (ablation) | TimeSformer · NTU datasets | top-1 accuracy: NTU120 CS: 90.6 -> 90.5; CSet: 91.6 -> 91.6; NTU60 CS: 93.0 -> 93.0; CV: 97.2 -> 97.0 | No clear effect |
| use 3D-SIM instead of the baseline alone (ablation) | TimeSformer · Toyota-Smarthome CS; NTU120 CS/CSet; NTU60 CS/CV | mCA or top-1 accuracy: 68.4 -> 71.4; 90.6 -> 91.8; 91.6 -> 92.7; 93.0 -> 94.0; 97.2 -> 97.8 | Helped |
| replace the FC module in 2D-SIM or 3D-SIM with MLP or Transformer (ablation) | π-ViT · Toyota-Smarthome CS and NTU60 CVS1 | mCA or top-1 accuracy: 2D-SIM, SH CS: 72.5 -> 69.9/68.4; NTU CVS1: 88.0 -> 88.74/89.1; 3D-SIM, SH CS: 71.4 -> 70.7/69.9; NTU CVS1: 90.61 -> 90. | Mixed |
| replace 3D-SIM feature alignment with traditional feature or logit distillation (ablation) | TimeSformer · Toyota-Smarthome CS and NTU60 CVS1 | mCA or top-1 accuracy: SH CS: 71.4 -> 67.1, 67.6, 68.3, 69.1; NTU CVS1: 90.6 -> 85.1, 84.4, 87.9, 86.7 | Hurt |
| use local or global+local rather than global alignment in 3D-SIM (ablation) | π-ViT · Toyota-Smarthome CS and NTU60 CVS1 | mCA or top-1 accuracy: SH CS position 12: 71.4 -> 70.6/71.3; NTU CVS1 position 12: 90.6 -> 90.3/90.1 | Mixed |
| replace the joint-specific token-skeleton map with a flat or depth variant (ablation) | 2D-SIM · Toyota-Smarthome CS and NTU60 CVS1 | mCA or top-1 accuracy: SH CS: 72.5 -> 71.1/71.3; NTU CVS1: 88.0 -> 87.8/87.8 | Hurt |
| remove the classification task from 3D-SIM (ablation) | π-ViT · Toyota-Smarthome CS and NTU60 CVS1 | mCA or top-1 accuracy: 71.4 -> 70.6; 90.3 -> 88.6; 70.6 -> 70.4; 90.6 -> 89.1 | Hurt |
| move 2D-SIM from position 1 to later positions or insert it at multiple positions (ablation) | π-ViT · Toyota-Smarthome CS and NTU60 CVS1 | mCA or top-1 accuracy: SH CS: 72.5 -> 70.9, 69.9, 70.7, 70.2; NTU CVS1: 88.0 -> 87.9, 87.5, 87.8, 87.7 | Hurt |
Surprising findings
- 2D-SIM does not improve action classification on NTU datasets, attributed to the lack of fine-grained appearance actions in lab settings.
- Traditional knowledge distillation (logit or feature, with class or distillation token) often underperforms or degrades the TimeSformer baseline, while 3D-SIM's alignment in a different embedding space helps.
Limitations
- Using 3D poses at inference for late fusion is needed to reach the best results, and PoseC3D outperforms π-ViT on NTU datasets.
- 2D-SIM gives no gain on controlled NTU datasets, so benefit is concentrated on real-world fine-grained ADL data.
Experimental setup
Datasets: Toyota-Smarthome NTU120 NTU60
Benchmarks: Toyota-Smarthome NTU120 NTU60
Metrics: mean class accuracy top-1 accuracy
Baselines: VPN++ PoseC3D Hyperformer 2s-AGCN Video Swin MotionFormer TimeSformer ViewCon STAR-Transformer 3D-Def-Transformer
Builds on: TimeSformer Hyperformer VPN++ knowledge distillation
Closest papers by meaning
- ExtPose: Robust and Coherent Pose Estimation by Extending ViTs (ICML 2025 · similarity 0.86)
- Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches (CVPR 2024 · similarity 0.85)
- SVTformer: Spatial-View-Temporal Transformer for Multi-View 3D Human Pose Estimation (AAAI 2025 · similarity 0.84)
- SKI Models: Skeleton Induced Vision-Language Embeddings for Understanding Activities of Daily Living (AAAI 2025 · similarity 0.84)
- HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation (CVPR 2025 · similarity 0.84)
- R^2-Art: Category-Level Articulation Pose Estimation from Single RGB Image via Cascade Render Strategy (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.