Research map / Video / Action recognition
Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DANCE, an ante-hoc concept-bottleneck video action recognizer whose concepts are disentangled into pose-sequence motion dynamics (clustered from key clips) and LLM-generated object/scene concepts, evaluated on KTH, Penn Action, HAA500 and UCF-101.
Problem
Explanations of video action recognition models are entangled saliency maps that do not separate motion dynamics from spatial context, while language-based concepts struggle to describe tacit motion.
Approach
Use three disjoint concept types in a concept bottleneck: human pose-sequence clusters for motion dynamics (label-free, via FINCH clustering of key-clip poses) and GPT-4o-proposed object and scene concepts pseudo-labeled with a vision-language dual encoder, with a frozen backbone and a sparse linear classifier on the concept activations.
Key results
- DANCE reaches 91.1% on KTH, 98.1% on Penn Action, 70.7% on HAA-100 and 87.5% on UCF-101, versus 89.7, 97.8, 73.5 and 88.4 for the baseline without interpretability.
- In a user study, more than 70% of responses favor DANCE explanations over each of three baselines; the motion dynamics concepts score 4.3/5 versus 2.3 for language concepts.
- On UCF-101-SCUBA, reweighting one concept raises accuracy from 84.0% to 86.5% without retraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DANCE's disentangled motion, object, and scene concepts instead of a non-interpretable baseline | Video action recognition · KTH | Top-1 accuracy (%): 89.7 -> 91.1 | Helped |
| Use DANCE's disentangled motion, object, and scene concepts instead of a non-interpretable baseline | Video action recognition · Penn Action | Top-1 accuracy (%): 97.8 -> 98.1 | Helped |
| Use DANCE's disentangled motion, object, and scene concepts instead of a non-interpretable baseline | Video action recognition · HAA-100 | Top-1 accuracy (%): 73.5 -> 70.7; drop of 2.8 points | Hurt |
| Use DANCE's disentangled motion, object, and scene concepts instead of a non-interpretable baseline | Video action recognition · UCF-101 | Top-1 accuracy (%): 88.4 -> 87.5; drop of 0.9 points | Hurt |
| Replace language-based or expert-defined temporal concepts with DANCE motion dynamics concepts | CBM · Motion concept user study | Average Likert score: 2.3 and 3.4 -> 4.3 | Helped |
| Use DANCE explanations instead of prior explainable video action recognition explanations | Video action recognition explanations · Pairwise user study | Responses preferring DANCE: more than 70% | Helped |
| Assign weight 1.0 to the relevant Volleyball Spiking motion concept | DANCE · UCF-101-SCUBA | Accuracy (%): 84.0% -> 86.5%; 2.5 points | Helped |
| Adjust class-level weights for Golf Swing and Tennis Swing | DANCE · UCF-101-SCUBA | Accuracy (%): 77.7% -> 82.0%; 4.3 point accuracy improvement | Helped |
Limitations
- Motion dynamics concepts rely on 2D pose estimation quality and filtering of low-confidence sequences.
- Object and scene concepts depend on LLM-generated candidates and a vision-language dual encoder for pseudo labels.
- Accuracy drops on HAA-100 (2.8 points) and UCF-101 (0.9 points) relative to the non-interpretable baseline.
Experimental setup
Datasets: KTH Penn Action HAA500 UCF-101
Benchmarks: KTH Penn Action HAA500 UCF-101 UCF-101-SCUBA
Metrics: Top-1 accuracy user study Likert score
Baselines: Baseline w/o interpretability CBM with UCF-101 attributes LF-CBM with entangled language concepts LF-CBM with disentangled language concepts GradCAM Saliency Tubes (VTCD)
Builds on: Concept Bottleneck Models Label-free CBM FINCH Saliency Tubes
Closest papers by meaning
- TRACE: Trajectory-Aware Conceps for Explainable Video Understanding (NeurIPS 2026 · similarity 0.90)
- DisMo: Disentangled Motion Representations for Open-World Motion Transfer (NeurIPS 2025 · similarity 0.88)
- Further Understanding Videos through Adverbs: A New Video Task (AAAI 2020 · similarity 0.88)
- Prompt-guided Disentangled Representation for Action Recognition (NeurIPS 2025 · similarity 0.87)
- HuMoCon: Concept Discovery for Human Motion Understanding (CVPR 2025 · similarity 0.87)
- Open-Vocabulary Video Relation Extraction (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.