Research map / Video / Video segmentation and tracking
Self-supervised video representation learning
Learns video features from temporal self-supervision, correspondence and reconstruction without labels.
167 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Video segmentation and tracking, keyword cluster “detection · anomaly · supervised”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Few recent reports The tracked benchmarks have at most one report from 2025–26 that could beat the old record, too few to say whether they are still moving.
42 papers in 2023–24 and 36 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 9.5% to 6.7%.
No longer reported by papers here: Something-Something V2 (Top-1 accuracy, last in 2023); Kinetics-600 (Top-1 accuracy, last in 2023); SSv2 (top-1 accuracy, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Learning Temporal Regularity in Video Sequences (CVPR 2016 · 1,422 citations)
- 2017 · Unsupervised Video Summarization With Adversarial LSTM Networks (CVPR 2017 · 637 citations)
- 2018 · Deep Reinforcement Learning for Unsupervised Video Summarization With Diversity-Representativeness Reward (AAAI 2018 · 478 citations)
- 2019 · Self-Supervised Spatio-Temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics (CVPR 2019 · 225 citations)
- 2020 · X3D: Expanding Architectures for Efficient Video Recognition (CVPR 2020 · 1,252 citations)
- 2021 · Spatiotemporal Contrastive Video Representation Learning (CVPR 2021 · 416 citations)
- 2022 · Masked Feature Prediction for Self-Supervised Visual Pre-Training (CVPR 2022 · 559 citations)
- 2023 · VideoMAE V2: Scaling Video Masked Autoencoders With Dual Masking (CVPR 2023 · 497 citations)
- 2024 · CSTA: CNN-based Spatiotemporal Attention for Video Summarization (CVPR 2024 · 28 citations)
- 2025 · Change3D: Revisiting Change Detection and Captioning from A Video Modeling Perspective (CVPR 2025 · 31 citations)
- 2026 · SAGA: Source Attribution of Generative AI Videos (CVPR 2026 · 1 citation)
Earliest papers here
- Learning Temporal Regularity in Video Sequences (CVPR 2016 · 1,422 citations)
- Hierarchical Recurrent Neural Encoder for Video Representation With Application to Captioning (CVPR 2016 · 427 citations)
- Highlight Detection With Pairwise Deep Ranking for First-Person Video Summarization (CVPR 2016 · 291 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
Kinetics-400 · top-1 accuracy (higher is better)
14 papers, 2021–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 85.4 | TokenLearner: Adaptive Space-Time Tokenization for Videos (NeurIPS 2021 · 66 citations) |
| 2022 | 87.4 | VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training (NeurIPS 2022 · 435 citations) |
| 2023 | 90.9 | Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning (CVPR 2023 · 67 citations) |
Something-Something V2 · Top-1 accuracy (higher is better)
9 papers, 2022–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 75.4 | VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training (NeurIPS 2022 · 435 citations) |
| 2023 | 77.3 | Masked Video Distillation: Rethinking Masked Feature Modeling for Self-Supervised Video Representation Learning (CVPR 2023 · 83 citations) |
HMDB51 · accuracy (higher is better)
8 papers, 2019–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 33.4 | Self-Supervised Spatio-Temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics (CVPR 2019 · 225 citations) |
| 2021 | 73.2 | CT-Net: Channel Tensorization Network for Video Classification (ICLR 2021 · 25 citations) |
| 2023 | 79.7 | Masked Video Distillation: Rethinking Masked Feature Modeling for Self-Supervised Video Representation Learning (CVPR 2023 · 83 citations) |
Kinetics-600 · Top-1 accuracy (higher is better)
7 papers, 2020–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 81.9 | X3D: Expanding Architectures for Efficient Video Recognition (CVPR 2020 · 1,252 citations) |
| 2021 | 86.3 | TokenLearner: Adaptive Space-Time Tokenization for Videos (NeurIPS 2021 · 66 citations) |
| 2022 | 88.3 | Masked Feature Prediction for Self-Supervised Visual Pre-Training (CVPR 2022 · 559 citations) |
| 2023 | 91.8 | Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning (CVPR 2023 · 67 citations) |
UCF101 · accuracy (higher is better)
7 papers, 2019–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 61.2 | Self-Supervised Spatio-Temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics (CVPR 2019 · 225 citations) |
| 2021 | 96.2 | CT-Net: Channel Tensorization Network for Video Classification (ICLR 2021 · 25 citations) |
| 2023 | 97.5 | Masked Video Distillation: Rethinking Masked Feature Modeling for Self-Supervised Video Representation Learning (CVPR 2023 · 83 citations) |
SSv2 · top-1 accuracy (higher is better)
5 papers, 2021–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 67.2 | Space-time Mixing Attention for Video Transformer (NeurIPS 2021 · 58 citations) |
| 2022 | 73.6 | Masked Autoencoders As Spatiotemporal Learners (NeurIPS 2022 · 241 citations) |
What has been tried
144 papers here report 1,644 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 5%, hurt 19%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add perception frames | HRSCD · Fscd | 69.48 -> 73.29 | Helped | Change3D: Revisiting Change Detection and Captioning from A Video Modeling Perspective (CVPR 2025 · 31 citations) |
| add StA without VB | CDF-v2 · AUC | 0.900 -> 0.926 | Helped | Generalizing Deepfake Video Detection with Plug-and-Play: Video-Level Blending and Spatiotemporal Adapter Tuning (CVPR 2025 · 28 citations) |
| Add the CNN Attention Module | SumMe · Spearman's ρ | 0.197 -> 0.205 | Helped | CSTA: CNN-based Spatiotemporal Attention for Video Summarization (CVPR 2024 · 28 citations) |
| Replace hierarchical encoder with isotropic CNXv2-S | DAVIS17 · J &Fm | 68.4 -> 53.8 | Hurt | VideoMAC: Video Masked Autoencoders Meet ConvNets (CVPR 2024 · 21 citations) |
| add pyramidal shortcuts with BSM to serial HNeRV | Parkour · PSNR | 25.07 -> 24.81; 26.56 -> 26.02; 24.34 -> 27.13 | Mixed | PNeRV: Enhancing Spatial Consistency via Pyramidal Neural Representation for Videos (CVPR 2024 · 16 citations) |
| add emotion-oriented masking | Ekman6 · AccE | 52.16 -> 53.17 | Helped | MART: Masked Affective RepresenTation Learning via Masked Temporal Distribution Distillation (CVPR 2024 · 15 citations) |
| Replace EAD with equal-frequency discretization | KuaiRec · MAE | 3.2150 -> 3.2477 | Hurt | CREAD: A Classification-Restoration Framework with Error Adaptive Discretization for Watch Time Prediction in Video Recommender Systems (AAAI 2024 · 14 citations) |
| use frame-level rather than clip-level OFL | UCF101 · R@1 | 80.85 -> 83.90 | Helped | No More Shortcuts: Realizing the Potential of Temporal Self-Supervision (AAAI 2024 · 9 citations) |
| Add adversarial training with the gradient reversal layer | UCF101 and HMDB51, 16/32/64 bits · GMAP | UCF101: 0.699 -> 0.719, 0.941 -> 0.959, 0.987 -> 1.09; HMDB51: 0.217 -> 0.233, 0.312 -> 0.338, 0.351 -> 0.376 | Helped | AutoSSVH: Exploring Automated Frame Sampling for Efficient Self-Supervised Video Hashing (CVPR 2025 · 8 citations) |
| add synthetic motion to pixel-target training | K400m · Top-1 accuracy (%) | 66.0 -> 68.1 | Helped | SMILE: Infusing Spatial and Motion Semantics in Masked Video Learning (CVPR 2025 · 8 citations) |
| use bidirectional Mamba blocks instead of forward-only or backward-only blocks | UCF101 and HMDB51, 16/32/64 bits · GmAP | 0.351, 0.375, 0.424; 0.079, 0.123, 0.136 (Forward Only); 0.324, 0.337, 0.390; 0.070, 0.120, 0.133 (Backward Only) | Helped | Efficient Self-Supervised Video Hashing with Selective State Spaces (AAAI 2025 · 7 citations) |
| add a past frame to the auxiliary branch | DAVIS-2017 · J &Fm | 61.9 -> 63.4 | Helped | When the Future Becomes the Past: Taming Temporal Correspondence for Self-supervised Video Representation Learning (CVPR 2025 · 7 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| VideoMAE | SSv2 | top-1 accuracy | 64.3 – 69.3 | 2 papers, e.g. ICML 2024 and CVPR 2023 |
| VideoMAE | Kinetics-400 | top-1 accuracy | 80.9 – 86.6 | 2 papers, e.g. CVPR 2023 and CVPR 2023 |
What reviewers will expect
Datasets and baselines named in the 18 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- Kinetics-400 · 7 of 18 papers
- HMDB51 · 6 of 18 papers
- Something-Something V2 · 5 of 18 papers
- ImageNet-1K · 5 of 18 papers
- UCF101 · 4 of 18 papers
- ActivityNet · 2 of 18 papers
- FCVID · 2 of 18 papers
- EPIC-Kitchens-100 · 2 of 18 papers
Baselines
- VideoMAE · 4 of 18 papers
- SSVH · 2 of 18 papers
- BTH · 2 of 18 papers
- DKPH · 2 of 18 papers
- MCMSH · 2 of 18 papers
- ConMH · 2 of 18 papers
- BerVAE · 2 of 18 papers
- MVD · 2 of 18 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 78 papers; from OpenAlex and the papers' affiliations).
- Peng Cheng Laboratory · 4 papers
- Tsinghua University · 4 papers
- Amazon · 4 papers
- Harbin Institute of Technology · 3 papers
- Peking University · 3 papers
- Huazhong University of Science and Technology · 3 papers
- Google DeepMind · 3 papers
- Nanjing University of Science and Technology · 2 papers
- Huawei · 2 papers
- Shanghai Jiao Tong University · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Video restoration with temporal priors (159 papers)
- Sign language and physiological video analysis (99 papers)
- Weakly supervised video anomaly detection (99 papers)
- Spatio-temporal predictive modeling (91 papers)
- Deepfake video detection (47 papers)
Elsewhere in the topic
- Layered video decomposition and dynamics (140 papers)
- Stochastic video prediction (125 papers)
- Physics-inspired motion prediction (113 papers)
- Matching-based video object segmentation (86 papers)
- Learned optical flow estimation (73 papers)
- Learned video compression (68 papers)
- Mask propagation for unsupervised video segmentation (67 papers)
- Video semantic and panoptic segmentation (67 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).