Research map / Video / Action recognition
Temporal action segmentation
Segments activities in untrimmed videos with unsupervised, weakly or semi-supervised clustering and transformers.
88 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Action recognition, keyword cluster “temporal action · action localization · action detection”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Few recent reports The tracked benchmarks have at most one report from 2025–26 that could beat the old record, too few to say whether they are still moving.
18 papers in 2023–24 and 16 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 12.8% to 12.1%.
No longer reported by papers here: Breakfast (MoF, last in 2024); 50Salads eval (MoF, last in 2022); Breakfast (IoU, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Actor-Action Semantic Segmentation With Grouping Process Models (CVPR 2016 · 41 citations)
- 2017 · UntrimmedNets for Weakly Supervised Action Recognition and Detection (CVPR 2017 · 586 citations)
- 2018 · NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning (CVPR 2018 · 143 citations)
- 2019 · Cross-Task Weakly Supervised Learning From Instructional Videos (CVPR 2019 · 202 citations)
- 2020 · End-to-End Learning of Visual Representations From Uncurated Instructional Videos (CVPR 2020 · 595 citations)
- 2021 · Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation (CVPR 2021 · 90 citations)
- 2022 · FineDiving: A Fine-Grained Dataset for Procedure-Aware Action Quality Assessment (CVPR 2022 · 143 citations)
- 2023 · Learning Procedure-Aware Video Representation From Instructional Videos and Their Narrations (CVPR 2023 · 23 citations)
- 2024 · FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation (CVPR 2024 · 52 citations)
- 2025 · Hierarchical Vector Quantization for Unsupervised Action Segmentation (AAAI 2025 · 13 citations)
- 2026 · Object-Centric Latent Action Learning (AAAI 2026 · 1 citation)
Earliest papers here
- Actor-Action Semantic Segmentation With Grouping Process Models (CVPR 2016 · 41 citations)
- Unsupervised Co-Activity Detection from Multiple Videos Using Absorbing Markov Chain (AAAI 2016 · 15 citations)
- UntrimmedNets for Weakly Supervised Action Recognition and Detection (CVPR 2017 · 586 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
Breakfast · MoF (higher is better)
7 papers, 2019–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 41.8 | Unsupervised Learning of Action Classes With Continuous Temporal Embedding (CVPR 2019 · 134 citations) |
| 2021 | 62.7 | Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation (CVPR 2021 · 90 citations) |
| 2022 | 75.2 | Iterative Contrast-Classify for Semi-supervised Temporal Action Segmentation (AAAI 2022 · 26 citations) |
Breakfast · Acc (higher is better)
7 papers, 2018–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 43 | NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning (CVPR 2018 · 143 citations) |
| 2021 | 64.1 | Temporal Action Segmentation From Timestamp Supervision (CVPR 2021 · 87 citations) |
| 2022 | 80 | Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos (CVPR 2022 · 77 citations) |
GTEA · Acc (higher is better)
5 papers, 2021–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 66.4 | Temporal Action Segmentation From Timestamp Supervision (CVPR 2021 · 87 citations) |
| 2022 | 81.2 | Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos (CVPR 2022 · 77 citations) |
| 2024 | 86.1 | FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation (CVPR 2024 · 52 citations) |
50Salads · Acc (higher is better)
4 papers, 2021–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 75.6 | Temporal Action Segmentation From Timestamp Supervision (CVPR 2021 · 87 citations) |
| 2022 | 88.1 | Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos (CVPR 2022 · 77 citations) |
50Salads eval · MoF (higher is better)
4 papers, 2019–2022 · not reported since 2022
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 60.6 | A Perceptual Prediction Framework for Self Supervised Event Segmentation (CVPR 2019 · 76 citations) |
Breakfast · IoU (higher is better)
4 papers, 2019–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 13.2 | Unsupervised Learning of Action Classes With Continuous Temporal Embedding (CVPR 2019 · 134 citations) |
| 2021 | 42.3 | Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation (CVPR 2021 · 90 citations) |
| 2024 | 44.4 | Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · 19 citations) |
What has been tried
79 papers here report 876 measured changes (a component added, removed or swapped, compared with a baseline): helped 75%, no clear effect 6%, hurt 16%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Remove action-branch supervision | Breakfast split 1 · F1@50 | 65.5 -> 40.0 | Hurt | FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation (CVPR 2024 · 52 citations) |
| add SAP | FineDiving-HM · ρ | 0.9351 -> 0.9435 | Helped | FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment (CVPR 2024 · 46 citations) |
| Initialize action clusters with k-means instead of random sampling | Breakfast, YouTube Instructions, 50 Salads Mid/Eval, Desktop Assembly · MoF | 56.1 -> 57.7; 52.9 -> 49.5; 46.2 -> 42.1; 59.3 -> 55.2; 70.4 -> 56.8 | Mixed | Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentation (CVPR 2024 · 27 citations) |
| Add action progress prediction | EgoProceL · F1@0.5 | 33.0 -> 35.2 | Helped | Progress-Aware Online Action Segmentation for Egocentric Procedural Task Videos (CVPR 2024 · 25 citations) |
| Add the context-dependent drop term | EgoProceL · EAE | 24.3 -> 13.3 | Helped | Learning to Predict Activity Progress by Self-Supervised Video Alignment (CVPR 2024 · 21 citations) |
| add the splitting step, rather than treating every frame as an actom | Breakfast · MoF | 54.8 -> 64.0 | Helped | Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · 19 citations) |
| add target-player descriptive words | FineSports · frame-mAP@0.5 | 20.67 -> 21.54 | Helped | FineSports: A Multi-person Hierarchical Sports Video Dataset for Fine-grained Action Understanding (CVPR 2024 · 18 citations) |
| Use hierarchical vector quantization instead of single-level quantization | Breakfast · F1 | 37.1 -> 39.7 | Helped | Hierarchical Vector Quantization for Unsupervised Action Segmentation (AAAI 2025 · 13 citations) |
| Train one joint model for actions and final states instead of separate models | ChangeIt held-out categories · final-state classification accuracy | 0.88 -> 0.79 | Hurt | GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos (CVPR 2024 · 12 citations) |
| add NoiseAug-based active learning selection to mean-teacher SSL | UCF101-24, 20% labels · v-mAP@0.5 | 72.2 -> 74.5 | Helped | Semi-supervised Active Learning for Video Action Detection (AAAI 2024 · 11 citations) |
| Use only class-agnostic boundary scores to select transitions | Breakfast · MoF | 67.9 -> 38.6 | Hurt | Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment (CVPR 2024 · 11 citations) |
| Add temporal coherence to generated replay segments | Breakfast · accuracy | 32.9 -> 41.8 | Helped | Coherent Temporal Synthesis for Incremental Action Segmentation (CVPR 2024 · 8 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| VTE-UNET | Breakfast | MoF | 48.1 – 52.2 | 2 papers, e.g. CVPR 2021 and CVPR 2021 |
What reviewers will expect
Datasets and baselines named in the 8 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- Breakfast · 2 of 8 papers
Baselines
- FACT · 2 of 8 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 34 papers; from OpenAlex and the papers' affiliations).
- Northeastern University · 4 papers
- National University of Singapore · 3 papers
- University of Central Florida · 2 papers
- The Chinese University of Hong Kong · 2 papers
- University of Science and Technology Beijing · 2 papers
- Peking University · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Weakly supervised temporal action localization (79 papers)
- Fully supervised temporal action detection (87 papers)
- Action anticipation and latent actions (79 papers)
Elsewhere in the topic
- Disentangled action recognition (182 papers)
- Spatiotemporal CNNs for action recognition (82 papers)
- Contrastive video representation learning (74 papers)
- Skeleton-based action recognition (46 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).