Research map / Video / Action recognition
Disentangled action recognition
Recognizes actions by decomposing dynamics, components and human-object interactions.
182 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Action recognition, keyword cluster “action recognition · video recognition · representation”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 3 of 6 tracked benchmarks.
33 papers in 2023–24 and 42 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 23.4% to 31.8%.
Open questions the data points to
- HMDB51 5-way 5-shot (accuracy): best went from 77.5 (up to 2024) to 85.5 in 2025 — Temporal Alignment-Free Video Matching for Few-shot Action Recognition (CVPR 2025 · 9 citations)
- UCF101 5-way 5-shot (accuracy): best went from 97.1 (up to 2024) to 98.8 in 2025 — Temporal Alignment-Free Video Matching for Few-shot Action Recognition (CVPR 2025 · 9 citations)
- Kinetics-400 (Top-1 accuracy): best went from 83.2 (up to 2024) to 88 in 2025 — Action Detail Matters: Refining Video Recognition with Local Action Queries (CVPR 2025 · 4 citations)
- Papers report MvPL on Kinetics-400 10% (Top-1 accuracy) anywhere from 36.9 to 58.2: comparisons against it depend on whose number you take.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · A Hierarchical Deep Temporal Model for Group Activity Recognition (CVPR 2016 · 539 citations)
- 2017 · Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset (CVPR 2017 · 9,707 citations)
- 2018 · Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition (CVPR 2018 · 349 citations)
- 2019 · Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (CVPR 2019 · 304 citations)
- 2020 · TEA: Temporal Excitation and Aggregation for Action Recognition (CVPR 2020 · 570 citations)
- 2021 · UAV-Human: A Large Benchmark for Human Behavior Understanding With Unmanned Aerial Vehicles (CVPR 2021 · 277 citations)
- 2022 · Spatio-Temporal Relation Modeling for Few-Shot Action Recognition (CVPR 2022 · 159 citations)
- 2023 · SVFormer: Semi-Supervised Video Transformer for Action Recognition (CVPR 2023 · 118 citations)
- 2024 · ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and More (CVPR 2024 · 24 citations)
- 2025 · Frame Order Matters: A Temporal Sequence-Aware Model for Few-Shot Action Recognition (AAAI 2025 · 11 citations)
- 2026 · Whole-Field Action Sensing via Wearable Single-Channel EMG Sensors and Resource-Efficient Motion Network (AAAI 2026 · 1 citation)
Earliest papers here
- A Hierarchical Deep Temporal Model for Group Activity Recognition (CVPR 2016 · 539 citations)
- A Key Volume Mining Deep Framework for Action Recognition (CVPR 2016 · 262 citations)
- Actions ~ Transformations (CVPR 2016 · 230 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
UCF101 · accuracy (higher is better)
10 papers, 2016–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2016 | 92.2 | VLAD3: Encoding Dynamics of Deep Features for Action Recognition (CVPR 2016 · 87 citations) |
| 2017 | 94.2 | Procedural Generation of Videos to Train Deep Action Recognition Networks (CVPR 2017 · 135 citations) |
| 2018 | 95.9 | Recognize Actions by Disentangling Components of Dynamics (CVPR 2018 · 102 citations) |
| 2019 | 96.5 | DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition (CVPR 2019 · 160 citations) |
| 2021 | 98.6 | SMART Frame Selection for Action Recognition (AAAI 2021 · 150 citations) |
HMDB-51 · accuracy (higher is better)
9 papers, 2016–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2016 | 66.9 | Discriminative Hierarchical Rank Pooling for Activity Recognition (CVPR 2016 · 115 citations) |
| 2017 | 69.5 | Procedural Generation of Videos to Train Deep Action Recognition Networks (CVPR 2017 · 135 citations) |
| 2019 | 77.8 | DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition (CVPR 2019 · 160 citations) |
| 2021 | 84.3 | SMART Frame Selection for Action Recognition (AAAI 2021 · 150 citations) |
UCF-101 · Top-1 accuracy (higher is better)
8 papers, 2019–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 93.6 | Temporal Bilinear Networks for Video Action Recognition (AAAI 2019 · 44 citations) |
| 2020 | 97.8 | Knowledge Integration Networks for Action Recognition (AAAI 2020 · 18 citations) |
HMDB51 5-way 5-shot · accuracy (higher is better)
7 papers, 2021–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 75.6 | Temporal-Relational CrossTransformers for Few-Shot Action Recognition (CVPR 2021 · 10 citations) |
| 2023 | 77.5 | Revisiting the Spatial and Temporal Modeling for Few-Shot Action Recognition (AAAI 2023 · 50 citations) |
| 2025 | 85.5 | Temporal Alignment-Free Video Matching for Few-shot Action Recognition (CVPR 2025 · 9 citations) |
UCF101 5-way 5-shot · accuracy (higher is better)
7 papers, 2021–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 96.1 | Temporal-Relational CrossTransformers for Few-Shot Action Recognition (CVPR 2021 · 10 citations) |
| 2023 | 97.1 | Revisiting the Spatial and Temporal Modeling for Few-Shot Action Recognition (AAAI 2023 · 50 citations) |
| 2025 | 98.8 | Temporal Alignment-Free Video Matching for Few-shot Action Recognition (CVPR 2025 · 9 citations) |
Kinetics-400 · Top-1 accuracy (higher is better)
6 papers, 2020–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 77.8 | Knowledge Integration Networks for Action Recognition (AAAI 2020 · 18 citations) |
| 2022 | 83.2 | Multi-dataset Training of Transformers for Robust Action Recognition (NeurIPS 2022 · 2 citations) |
| 2025 | 88 | Action Detail Matters: Refining Video Recognition with Local Action Queries (CVPR 2025 · 4 citations) |
What has been tried
154 papers here report 1,784 measured changes (a component added, removed or swapped, compared with a baseline): helped 75%, no clear effect 6%, hurt 17%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Train ExACT with caption-level rather than category-level labels | SeAct · Top-1 accuracy | 66.07 -> 67.24 | Helped | ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and More (CVPR 2024 · 24 citations) |
| Use learned positional encoding instead of interval encoding | EPIC-KITCHENS-100 · Action top-1 accuracy | 57.5 -> 29.6 | Hurt | TIM: A Time Interval Machine for Audio-Visual Action Recognition (CVPR 2024 · 22 citations) |
| add 2D-SIM instead of using the baseline alone | NTU datasets · top-1 accuracy | NTU120 CS: 90.6 -> 90.5; CSet: 91.6 -> 91.6; NTU60 CS: 93.0 -> 93.0; CV: 97.2 -> 97.0 | No clear effect | Just Add ?! Pose Induced Video Transformers for Understanding Activities of Daily Living (CVPR 2024 · 21 citations) |
| Add the entity text corpus instead of using action-name text embeddings | HMDB-51 2-shot · top-1 accuracy | 55.2 -> 58.1 | Helped | Align Before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition (CVPR 2024 · 14 citations) |
| add the graph module (grh) | WTT table tennis dataset · Accuracy | 82.0 -> 83.5 | Helped | ViSTec: Video Modeling for Sports Technique Recognition and Tactical Analysis (AAAI 2024 · 12 citations) |
| Replace motion-consistent noise with iid noise | IPN, KTH, and SBU · Action accuracy | Hurt | Selective Interpretable and Motion Consistent Privacy Attribute Obfuscation for Action Recognition (CVPR 2024 · 12 citations) | |
| Add Sequential Perceiver | SSv2-Small 1-shot · accuracy | 54.6 -> 58.2 | Helped | Frame Order Matters: A Temporal Sequence-Aware Model for Few-Shot Action Recognition (AAAI 2025 · 11 citations) |
| Add the Inner Module | SSv2 1-shot · accuracy | 46.3 -> 55.5 | Helped | Manta: Enhancing Mamba for Few-Shot Action Recognition of Long Sub-Sequence (AAAI 2025 · 9 citations) |
| Add LWA and SWA to GCN+Trans. | P2A (joint modality) · accuracy | 58.4 -> 69.3 | Helped | VA-AR: Learning Velocity-Aware Action Representations with Mixture of Window Attention (AAAI 2025 · 9 citations) |
| Add exclusive pattern tokens | HMDB51, Kinetics, and UCF101 5-way 1-shot · accuracy | 61.8 -> 62.5; 74.6 -> 75.0; 86.7 -> 86.8 | Helped | Temporal Alignment-Free Video Matching for Few-shot Action Recognition (CVPR 2025 · 9 citations) |
| add feature representation loss L_recon | UCF101 · Top-1 accuracy | 64.1 -> 65.2 | Helped | Generative Model-Based Feature Knowledge Distillation for Action Recognition (AAAI 2024 · 9 citations) |
| Increase sampled frames from 2 to 16 | ActionAtlas · accuracy | 24.07 -> 30.24 | Helped | ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition (NeurIPS 2024 · 7 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| STRM | HMDB51, 5-way 5-shot | accuracy | 77.3 – 83.2 | 3 papers, e.g. AAAI 2023 and CVPR 2025 |
| MvPL | Kinetics-400 10% | Top-1 accuracy | 36.9 – 58.2 | 2 papers, e.g. CVPR 2022 and CVPR 2023 |
| MvPL | UCF-101 10% | Top-1 accuracy | 55.5 – 80.5 | 2 papers, e.g. CVPR 2022 and CVPR 2023 |
| OR2G | Charades validation | mAP | 44.9 – 63.3 | 2 papers, e.g. CVPR 2024 and NeurIPS 2025 |
| OTAM | Kinetics 5-way 1-shot | accuracy | 73 – 82.2 | 2 papers, e.g. CVPR 2022 and CVPR 2025 |
| OTAM | Kinetics 5-way 5-shot | accuracy | 85.8 – 91.6 | 2 papers, e.g. CVPR 2021 and CVPR 2025 |
What reviewers will expect
Datasets and baselines named in the 19 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- Kinetics-400 · 7 of 19 papers
- UCF101 · 7 of 19 papers
- HMDB-51 · 6 of 19 papers
- Something-Something v2 · 4 of 19 papers
- Kinetics · 3 of 19 papers
- Charades · 3 of 19 papers
- SSv2-Small · 2 of 19 papers
- ImageNet · 2 of 19 papers
Baselines
- ActionCLIP · 4 of 19 papers
- X-CLIP · 4 of 19 papers
- OST · 3 of 19 papers
- TRX · 3 of 19 papers
- HyRSM · 3 of 19 papers
- MoLo · 3 of 19 papers
- ST-Adapter · 3 of 19 papers
- Vita-CLIP · 3 of 19 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 75 papers; from OpenAlex and the papers' affiliations).
- Zhejiang University · 4 papers
- Chinese Academy of Sciences · 3 papers
- Massachusetts Institute of Technology · 3 papers
- University of Central Florida · 3 papers
- Huawei · 3 papers
- The Chinese University of Hong Kong · 2 papers
- Beijing University of Posts and Telecommunications · 2 papers
- Goethe University Frankfurt · 2 papers
- MIT-IBM Watson AI Lab · 2 papers
- Xidian University · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Spatiotemporal CNNs for action recognition (82 papers)
- Contrastive video representation learning (74 papers)
- Skeleton-based action recognition (46 papers)
Elsewhere in the topic
- Temporal action segmentation (88 papers)
- Fully supervised temporal action detection (87 papers)
- Weakly supervised temporal action localization (79 papers)
- Action anticipation and latent actions (79 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).