Research map / Video / Action recognition
Prompt-guided Disentangled Representation for Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProDA, which uses a Dynamic Prompt Module and a video-adapted graph parsing network over spatio-temporal scene graphs to disentangle any specified actions from multi-action videos, evaluated on Charades and SportsHHI.
Problem
Action recognition in multi-action videos struggles because unified features mix all actions, making it hard to model interactions between objects for specific actions.
Approach
Prompt-conditioned disentanglement: a Dynamic Prompt Module builds node-adaptive prompts from multi-hot specified/unspecified action vectors and node features, which steer a video-adapted GPNN (VGPNN) with VGNorm and an AD loss (Pearson-correlation disentanglement plus margin-based reconstruction) to split the SSG into specified and remaining-action parts.
Key results
- On Charades validation, Ours (B/16, CLIP, RGB+SSG) reaches 71.1 mAP, above VicTR (L/14) at 57.6 and OR2G+NL at 67.5.
- On SportsHHI, the baseline plus ProDA reaches 13.55 mAP vs 10.69 for the official baseline and 10.16 for the parameter-matched reimplementation.
- Ablation on Charades: DPM improves a_t mAP from 68.07 (SimplePrompt) to 71.10; VGNorm beats LayerNorm, BatchNorm and GraphNorm (71.10 vs 68.61 best other).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SimplePrompt with DPM (ablation) | ProDA · Charades | a_t mAP: 68.07 -> 71.10 | Helped |
| Replace SimplePrompt with DPM (ablation) | ProDA · Charades | a_m mAP: 57.80 -> 59.04 | Helped |
| Use VGNorm instead of BatchNorm (ablation) | ProDA · Charades | a_t mAP: +5.71\% mAP | Helped |
| Use VGNorm instead of LayerNorm (ablation) | ProDA · Charades | a_t mAP: +2.98\% mAP | Helped |
| Use VGNorm instead of GraphNorm (ablation) | ProDA · Charades | a_t mAP: +2.49\% mAP | Helped |
| Remove the reconstruction and disentanglement margins (ablation) | ProDA · Charades | a_m mAP: 57.38 -> 8.03 | Hurt |
| Add disentanglement loss without reconstruction loss (ablation) | ProDA · Charades | a_t mAP: 67.45 -> 67.78 | Helped |
| Add reconstruction loss without disentanglement loss (ablation) | ProDA · Charades | a_t mAP: 67.45 -> 64.89 | Hurt |
| Combine reconstruction and disentanglement losses with margins (ablation) | ProDA · Charades | a_t mAP: 67.45 -> 71.10 | Helped |
| Add ProDA to the SportsHHI baseline | SportsHHI baseline · SportsHHI | mAP: 10.69 -> 13.55 | Helped |
| Add ProDA to the SportsHHI baseline | SportsHHI baseline · SportsHHI | R@150: 89.25 -> 92.54 | Helped |
| Add ProDA to the SportsHHI baseline | SportsHHI baseline · SportsHHI | R@100: 82.93 -> 87.91 | Helped |
Surprising findings
- Removing the margin in the disentanglement/reconstruction losses makes the auxiliary pre-disentanglement prediction a_m collapse to 8.03 mAP, suggesting the prompt alone drives the output (shortcut learning).
Limitations
- Operates only on visual features and derived SSGs, without audio, text, or pose modalities.
- Relies on ground-truth or pre-generated SSGs and an off-the-shelf video encoder.
Experimental setup
Datasets: Charades Action Genome SportsHHI MultiSports
Benchmarks: Charades SportsHHI
Metrics: mAP recall at K
Baselines: SGFB+NL I3D+NL OR2G+NL LaIAR+NL VicTR STTran HORT SlowFast ACARN SportsHHI
Builds on: Graph Parsing Neural Network (GPNN) Dual-AI GraphNorm VicTR
Closest papers by meaning
- Recognize Actions by Disentangling Components of Dynamics (CVPR 2018 · similarity 0.88)
- Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition (NeurIPS 2025 · similarity 0.87)
- DVANet: Disentangling View and Action Features for Multi-View Action Recognition (AAAI 2024 · similarity 0.87)
- Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP (AAAI 2025 · similarity 0.86)
- DisMo: Disentangled Motion Representations for Open-World Motion Transfer (NeurIPS 2025 · similarity 0.86)
- Action Detail Matters: Refining Video Recognition with Local Action Queries (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.