Research map / Label-efficient and robust learning / Recognition and long-tail
DVANet: Disentangling View and Action Features for Multi-View Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DVANet, a transformer-decoder model with separate learnable action and view queries plus two supervised contrastive losses to disentangle view and action features for RGB-only multi-view action recognition.
Problem
Multi-view action recognition is hard because viewpoint changes (background, occlusion, visibility) entangle view-relevant information with action features, degrading recognition accuracy.
Approach
Use a multi-query transformer decoder over a 3D-CNN (R3D) encoder with N_a action queries and one view query, then enforce disentanglement with two opposing supervised contrastive losses built from anchor, same-action-different-view, and same-view-different-action triplets, plus an orthogonal query loss.
Key results
- On NTU RGB+D, DVANet reaches 93.4% cross-subject and 98.2% cross-view, a +1.5% gain over the best RGB method in cross-subject.
- On NTU RGB+D 120, DVANet gets 90.4% cross-subject, +4.8% over the best prior RGB method.
- On PKU-MMD cross-subject, DVANet reaches 95.8%, and removing both contrastive losses drops accuracy by nearly 2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add view cross-entropy and view contrastive losses (ablation) | DVANet · PKU-MMD cross-subject | accuracy: 94.0 -> 95.8 | Helped |
| Add action contrastive loss (ablation) | DVANet · PKU-MMD cross-subject | accuracy: 95.3 -> 95.8 | Helped |
| Add view contrastive loss (ablation) | DVANet · PKU-MMD cross-subject | accuracy: 95.2 -> 95.8 | Helped |
| Add action and view contrastive losses (ablation) | DVANet · PKU-MMD cross-subject | accuracy: 94.2 -> 95.8 | Helped |
| Add orthogonal query loss (ablation) | DVANet · PKU-MMD cross-subject | accuracy: 95.0 -> 95.8 | Helped |
Surprising findings
- Skeleton-based methods remain strong baselines, yet the RGB-only DVANet beats them on NTU RGB+D cross-subject and cross-view.
- Training from scratch on N-UCLA reportedly outperforms prior RGB works, which often required transfer learning.
Limitations
- Ablations are run only on PKU-MMD cross-subject, so generality of loss contributions is not shown on other datasets.
- Model is trained in a supervised manner for view-invariant recognition from seen viewpoints; unseen-view performance is only shown qualitatively in supplement.
Experimental setup
Datasets: NTU RGB+D NTU RGB+D 120 PKU-MMD N-UCLA
Benchmarks: NTU RGB+D NTU RGB+D 120 PKU-MMD N-UCLA
Metrics: cross-subject accuracy cross-view accuracy cross-setup accuracy
Baselines: MMNet DG-STGCN EfficientGCN PKU-MMD multimodal baselines (RGB+Depth TSN, TSM)
Builds on: R3D supervised contrastive learning DETR-style transformer decoder queries
Closest papers by meaning
- Recognize Actions by Disentangling Components of Dynamics (CVPR 2018 · similarity 0.90)
- Unsupervised Learning of View-invariant Action Representations (NeurIPS 2018 · similarity 0.88)
- Recognizing Actions in Videos From Unseen Viewpoints (CVPR 2021 · similarity 0.88)
- Learning View-Disentangled Human Pose Representation by Contrastive Cross-View Mutual Information Maximization (CVPR 2021 · similarity 0.87)
- Multi-dataset Training of Transformers for Robust Action Recognition (NeurIPS 2022 · similarity 0.87)
- Prompt-guided Disentangled Representation for Action Recognition (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.