Research map / Video / Action recognition
Spatiotemporal CNNs for action recognition
Designs 3D and residual convolutional architectures for video action recognition.
82 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2023) · an approach in Action recognition, keyword cluster “action recognition · video recognition · representation”
State History Benchmarks Tried Baselines Preprints Who Export
Where it stands
Quiet None of the tracked benchmarks has a report from 2025–26.
4 papers in 2023–24 and 0 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 2.8% to 0.0%.
No longer reported by papers here: Kinetics-400 (top-1 accuracy, last in 2022); UCF101 (accuracy, last in 2021); HMDB51 (accuracy, last in 2021); Something-Something V2 (Top-1 accuracy, last in 2022); Something-Something V1 (top-1 accuracy (%), last in 2022); UCF101 (split 1) (accuracy, last in 2020).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Convolutional Two-Stream Network Fusion for Video Action Recognition (CVPR 2016 · 2,806 citations)
- 2017 · Spatiotemporal Multiplier Networks for Video Action Recognition (CVPR 2017 · 713 citations)
- 2018 · A Closer Look at Spatiotemporal Convolutions for Action Recognition (CVPR 2018 · 3,632 citations)
- 2019 · MARS: Motion-Augmented RGB Stream for Action Recognition (CVPR 2019 · 295 citations)
- 2020 · Temporal Pyramid Network for Action Recognition (CVPR 2020 · 469 citations)
- 2021 · TDN: Temporal Difference Networks for Efficient Action Recognition (CVPR 2021 · 489 citations)
- 2022 · Video Swin Transformer (CVPR 2022 · 2,080 citations)
- 2023 · A Large-Scale Robustness Analysis of Video Action Recognition Models (CVPR 2023 · 29 citations)
Earliest papers here
- Convolutional Two-Stream Network Fusion for Video Action Recognition (CVPR 2016 · 2,806 citations)
- Generating Videos with Scene Dynamics (NeurIPS 2016 · 837 citations)
- Spatiotemporal Residual Networks for Video Action Recognition (NeurIPS 2016 · 657 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
Kinetics-400 · top-1 accuracy (higher is better)
17 papers, 2019–2022 · not reported since 2022
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 73.5 | More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation (NeurIPS 2019 · 96 citations) |
| 2020 | 81 | Video Modeling With Correlation Networks (CVPR 2020 · 164 citations) |
| 2021 | 83 | Shifted Chunk Transformer for Spatio-Temporal Representational Learning (NeurIPS 2021 · 7 citations) |
| 2022 | 89.1 | Multiview Transformers for Video Recognition (CVPR 2022 · 292 citations) |
UCF101 · accuracy (higher is better)
17 papers, 2016–2021 · not reported since 2021
| Year | Best so far | Set by |
|---|---|---|
| 2016 | 89.1 | Dynamic Image Networks for Action Recognition (CVPR 2016 · 606 citations) |
| 2017 | 95.6 | Deep Temporal Linear Encoding Networks (CVPR 2017 · 254 citations) |
| 2018 | 97.3 | A Closer Look at Spatiotemporal Convolutions for Action Recognition (CVPR 2018 · 3,632 citations) |
| 2021 | 97.4 | TDN: Temporal Difference Networks for Efficient Action Recognition (CVPR 2021 · 489 citations) |
HMDB51 · accuracy (higher is better)
16 papers, 2016–2021 · not reported since 2021
| Year | Best so far | Set by |
|---|---|---|
| 2016 | 65.2 | Dynamic Image Networks for Action Recognition (CVPR 2016 · 606 citations) |
| 2017 | 72.2 | Spatiotemporal Multiplier Networks for Video Action Recognition (CVPR 2017 · 713 citations) |
| 2018 | 78.7 | A Closer Look at Spatiotemporal Convolutions for Action Recognition (CVPR 2018 · 3,632 citations) |
Something-Something V2 · Top-1 accuracy (higher is better)
10 papers, 2020–2022 · not reported since 2022
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 61.2 | Regularization on Spatio-Temporally Smoothed Feature for Action Recognition (CVPR 2020 · 31 citations) |
| 2021 | 68.2 | TDN: Temporal Difference Networks for Efficient Action Recognition (CVPR 2021 · 489 citations) |
| 2022 | 69.5 | Object-Region Video Transformers (CVPR 2022 · 75 citations) |
Something-Something V1 · top-1 accuracy (%) (higher is better)
6 papers, 2020–2022 · not reported since 2022
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 50.4 | V4D: 4D Convolutional Neural Networks for Video-level Representation Learning (ICLR 2020 · 48 citations) |
| 2021 | 55.1 | TDN: Temporal Difference Networks for Efficient Action Recognition (CVPR 2021 · 489 citations) |
UCF101 (split 1) · accuracy (higher is better)
6 papers, 2016–2020 · not reported since 2020
| Year | Best so far | Set by |
|---|---|---|
| 2016 | 90.4 | Convolutional Two-Stream Network Fusion for Video Action Recognition (CVPR 2016 · 2,806 citations) |
| 2017 | 94.2 | Spatiotemporal Pyramid Network for Video Action Recognition (CVPR 2017 · 287 citations) |
What has been tried
79 papers here report 958 measured changes (a component added, removed or swapped, compared with a baseline): helped 76%, no clear effect 5%, hurt 16%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| SlowFast | Kinetics-400 | top-1 accuracy | 75 – 79.8 | 4 papers, e.g. CVPR 2020 and NeurIPS 2021 |
| C3D | HMDB51 | accuracy | 56.8 – 62.1 | 2 papers, e.g. CVPR 2017 and AAAI 2019 |
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 4 papers; from OpenAlex and the papers' affiliations).
- Microsoft · 2 papers
Use it
Download BibTeX of 20 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Disentangled action recognition (182 papers)
- Contrastive video representation learning (74 papers)
- Skeleton-based action recognition (46 papers)
Elsewhere in the topic
- Temporal action segmentation (88 papers)
- Fully supervised temporal action detection (87 papers)
- Weakly supervised temporal action localization (79 papers)
- Action anticipation and latent actions (79 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).