Research map / Video / Action recognition
Contrastive video representation learning
Learns video features with contrastive objectives, continuity, motion decoupling and language concepts.
74 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Action recognition, keyword cluster “action recognition · video recognition · representation”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Few recent reports The tracked benchmarks have at most one report from 2025–26 that could beat the old record, too few to say whether they are still moving.
15 papers in 2023–24 and 6 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 10.6% to 4.5%.
No longer reported by papers here: HMDB51 (top-1 accuracy, last in 2023); UCF101 (top-1 accuracy, last in 2023); HMDB51 (accuracy, last in 2022); UCF101 (accuracy, last in 2022); FCVID (mAP, last in 2022).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Anticipating Visual Representations From Unlabeled Video (CVPR 2016 · 543 citations)
- 2017 · Self-Supervised Video Representation Learning With Odd-One-Out Networks (CVPR 2017 · 454 citations)
- 2018 · Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (NeurIPS 2018 · 331 citations)
- 2019 · Self-Supervised Spatiotemporal Learning via Video Clip Order Prediction (CVPR 2019 · 446 citations)
- 2020 · Few-Shot Video Classification via Temporal Alignment (CVPR 2020 · 273 citations)
- 2021 · Is Space-Time Attention All You Need for Video Understanding? (ICML 2021 · 1,464 citations)
- 2022 · MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition (CVPR 2022 · 171 citations)
- 2023 · Vita-CLIP: Video and Text Adaptive CLIP via Multimodal Prompting (CVPR 2023 · 111 citations)
- 2024 · OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition (CVPR 2024 · 10 citations)
- 2025 · Beyond Image Classification: A Video Benchmark and Dual-Branch Hybrid Discrimination Framework for Compositional Zero-Shot Learning (CVPR 2025 · 3 citations)
- 2026 · Privacy Beyond Pixels: Latent Anonymization for Privacy-Preserving Video Understanding (ICLR 2026 · 0 citations)
Earliest papers here
- Anticipating Visual Representations From Unlabeled Video (CVPR 2016 · 543 citations)
- Slow and Steady Feature Analysis: Higher Order Temporal Coherence in Video (CVPR 2016 · 153 citations)
- You Lead, We Exceed: Labor-Free Video Concept Learning by Jointly Exploiting Web Videos and Images (CVPR 2016 · 123 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
Kinetics-400 · Top-1 accuracy (higher is better)
9 papers, 2021–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 82.3 | Boosting Video Representation Learning With Multi-Faceted Integration (CVPR 2021 · 7 citations) |
HMDB51 · top-1 accuracy (higher is better)
8 papers, 2019–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 33.7 | Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles (AAAI 2019 · 346 citations) |
| 2020 | 68.9 | Self-Supervised Learning by Cross-Modal Audio-Video Clustering (NeurIPS 2020 · 248 citations) |
UCF101 · top-1 accuracy (higher is better)
8 papers, 2019–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 65.8 | Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles (AAAI 2019 · 346 citations) |
| 2020 | 95.5 | Self-Supervised Learning by Cross-Modal Audio-Video Clustering (NeurIPS 2020 · 248 citations) |
HMDB51 · accuracy (higher is better)
5 papers, 2017–2022 · not reported since 2022
| Year | Best so far | Set by |
|---|---|---|
| 2017 | 32.4 | Self-Supervised Video Representation Learning With Odd-One-Out Networks (CVPR 2017 · 454 citations) |
| 2018 | 56.9 | Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (NeurIPS 2018 · 331 citations) |
| 2022 | 75.9 | Long-Short Temporal Contrastive Learning of Video Transformers (CVPR 2022 · 46 citations) |
UCF101 · accuracy (higher is better)
5 papers, 2017–2022 · not reported since 2022
| Year | Best so far | Set by |
|---|---|---|
| 2017 | 60 | Self-Supervised Video Representation Learning With Odd-One-Out Networks (CVPR 2017 · 454 citations) |
| 2018 | 89 | Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (NeurIPS 2018 · 331 citations) |
| 2022 | 96.8 | Long-Short Temporal Contrastive Learning of Video Transformers (CVPR 2022 · 46 citations) |
FCVID · mAP (higher is better)
4 papers, 2019–2022 · not reported since 2022
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 80.2 | AdaFrame: Adaptive Frame Selection for Fast Video Recognition (CVPR 2019 · 232 citations) |
| 2022 | 84.4 | Rethinking Resolution in the Context of Efficient Video Recognition (NeurIPS 2022 · 10 citations) |
What has been tried
68 papers here report 805 measured changes (a component added, removed or swapped, compared with a baseline): helped 74%, no clear effect 5%, hurt 18%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Replace category-name matching with pooled descriptors | HMDB · Top-1 accuracy | 50.9 -> 53.3 (+2.4) | Helped | OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition (CVPR 2024 · 10 citations) |
| Generate tubelet proposals with feature-space SLIC rather than random crops | TCOW Kubric · Positive perturbation | 0.174 -> 0.102 | Helped | Understanding Video Transformers via Universal Concept Discovery (CVPR 2024 · 8 citations) |
| Add the object-first branch to the condition-first branch | C-EgoExo · HM | 15.1 -> 17.5 | Helped | Beyond Image Classification: A Video Benchmark and Dual-Branch Hybrid Discrimination Framework for Compositional Zero-Shot Learning (CVPR 2025 · 3 citations) |
| Add RST feature translator | Ego-Exo4D cross-view retrieval, ego→exo · HR@5 | 44.4 -> 58.1 | Helped | Viewpoint Rosetta Stone: Unlocking Unpaired Ego-Exo Videos for View-invariant Representation Learning (CVPR 2025 · 2 citations) |
| Add a 2-layer Transformer head without GRW smoothing | Kinetics-600 · Top-1 accuracy | +0.9 Top-1 | Helped | Smooth Regularization for Efficient Video Recognition (NeurIPS 2025 · 0 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| AR-Net | ActivityNet-v1.3 | mAP (%) | 73.8 – 79.7 | 2 papers, e.g. NeurIPS 2022 and CVPR 2021 |
What reviewers will expect
Datasets and baselines named in the 5 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- EgoExo-4D · 2 of 5 papers
- Kinetics-400 · 2 of 5 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 21 papers; from OpenAlex and the papers' affiliations).
- LG AI Research · 2 papers
- Shanghai AI Laboratory · 2 papers
- University of Central Florida · 2 papers
- Meta · 2 papers
Use it
Download BibTeX of 28 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Disentangled action recognition (182 papers)
- Spatiotemporal CNNs for action recognition (82 papers)
- Skeleton-based action recognition (46 papers)
Elsewhere in the topic
- Temporal action segmentation (88 papers)
- Fully supervised temporal action detection (87 papers)
- Weakly supervised temporal action localization (79 papers)
- Action anticipation and latent actions (79 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).