Research map / Video / Action recognition
Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SaM, a training-free unsupervised action segmentation method that splits videos into actoms via subspace-based cosine similarity and local minimum search, then merges actoms with a spatio-temporal similarity.
Problem
Unsupervised action segmentation in untrimmed videos typically requires training on the target dataset with pseudo-labels, which is slow and noisy; the paper aims for an accurate method that needs no training.
Approach
Treat frames as lying in distinct subspaces on a hypersphere, detect actom boundaries as local minima of cosine-based adjacent-frame similarity, then iteratively merge actoms using a spatio-temporal similarity that combines feature cosine similarity with temporal proximity, with no learned parameters.
Key results
- On Breakfast, SaM reaches 64.0 MoF, 12 points above the best reported unsupervised method CoSeg (53.1 MoF).
- On YouTube Instructional Videos, SaM reaches 68.1 MoF and 49.6 F1, gaining 15.7 MoF over LTL.
- Learning time is 0.20 s per video on Breakfast, about 56x faster than CTE, with no training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SaM instead of CoSeg | Unsupervised action segmentation · Breakfast | MoF: 53.1 -> 64.0 | Helped |
| use SaM instead of LTL | Unsupervised action segmentation · YouTube Instructional Videos | F1: 34.7 -> 49.6 | Helped |
| use SaM instead of LTL | Unsupervised action segmentation · YouTube Instructional Videos | MoF: 52.4 -> 68.1 | Helped |
| use SaM instead of SRL | Unsupervised action segmentation · Hollywood Extended | IoU: 30.0 -> 39.4 | Helped |
| use SaM instead of DMR | Unsupervised action segmentation · Hollywood Extended | F1: 47.4 -> 57.3 | Helped |
| use SaM instead of DHC | Unsupervised action segmentation · Hollywood Extended | MoF: 55.2 -> 60.9 | Helped |
| use SaM instead of SSTDA | Unsupervised action segmentation · 50Salads | F1: 73.8 -> 78.2 | Helped |
| use SaM instead of C2F2 | Unsupervised action segmentation · 50Salads (mid granularity) | MoF: 56.2 -> 71.9 | Helped |
| add the splitting step, rather than treating every frame as an actom (ablation) | SaM · Breakfast | MoF: 54.8 -> 64.0 | Helped |
| add the splitting step, rather than treating every frame as an actom (ablation) | SaM · Breakfast | IoU: 35.1 -> 44.4 | Helped |
| add the splitting step, rather than treating every frame as an actom (ablation) | SaM · Breakfast | F1: 39.6 -> 55.9 | Helped |
| add the merging step instead of keeping only K − 1 boundaries (ablation) | SaM · Breakfast | MoF: 50.6 -> 64.0 | Helped |
Surprising findings
- Removing the splitting step drops Breakfast MoF from 64.0 to 54.8, and removing merging drops it to 50.6.
Limitations
- Performance on 50Salads is noted as lower than fully supervised methods and leaves room for improvement.
- Increasing the temporal weight lambda can wrongly merge adjacent distinct actions (e.g., A, B, A merged).
Experimental setup
Datasets: Breakfast YouTube Instructional Videos Hollywood Extended 50Salads
Benchmarks: Breakfast YouTube Instructional Videos Hollywood Extended 50Salads
Metrics: MoF F1 IoU training and learning time
Baselines: CTE CoSeg ASAL TOT+TCL LTL C2F MuCon CDFL SRL DHC
Builds on: Menon et al. 2020 bottom-up hypersphere partitioning Improved Dense Trajectories
Closest papers by meaning
- FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation (CVPR 2024 · similarity 0.89)
- Hierarchical Vector Quantization for Unsupervised Action Segmentation (AAAI 2025 · similarity 0.89)
- ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation (NeurIPS 2024 · similarity 0.89)
- Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment (CVPR 2024 · similarity 0.89)
- HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios (NeurIPS 2025 · similarity 0.88)
- OnlineTAS: An Online Baseline for Temporal Action Segmentation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.