Research map / Video / Action recognition
Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ASOT, a fused unbalanced Gromov-Wasserstein optimal transport post-processing method that yields temporally consistent action segmentations without known action order, and uses its pseudo-labels for unsupervised self-training.
Problem
Unsupervised temporal action segmentation of long untrimmed videos requires temporally consistent per-frame predictions, but prior OT-based methods assume balanced action distributions and a fixed known action ordering.
Approach
Encode temporal consistency as a Gromov-Wasserstein structural term fused with a visual cost matrix, and relax the marginal constraint on actions via KL penalty (fused unbalanced GW OT), solved efficiently with projected mirror descent; used both for pseudo-labelling during self-training and for decoding.
Key results
- On unsupervised (full) evaluation, ASOT attains 56.1 MoF on Breakfast and 70.4 MoF on Desktop Assembly, with comparable results to ASAL and UFSA (T) on Breakfast and major gains over UFSA (T) on FS (Mid) and DA.
- Applied as post-processing to MS-TCN++ on 50 Salads, MS + ASOT improves F1@10 from 80.1 to 84.4 and edit distance from 72.6 to 77.6.
- The largest video (16k frames, 19 classes) is solved in 26.1 ms on an RTX 4090 GPU, typically converging within 25 iterations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize action clusters with k-means instead of random sampling (ablation) | ASOT unsupervised training · Breakfast, YouTube Instructions, 50 Salads Mid/Eval, Desktop Assembly | MoF: 56.1 -> 57.7; 52.9 -> 49.5; 46.2 -> 42.1; 59.3 -> 55.2; 70.4 -> 56.8 | Mixed |
| Remove the global temporal prior (ablation) | ASOT unsupervised training · Breakfast, YouTube Instructions, 50 Salads Mid/Eval, Desktop Assembly | MoF: 56.1 -> 48.5; 52.9 -> 46.9; 46.2 -> 43.1; 59.3 -> 59.7; 70.4 -> 31.4 | Mixed |
| Force balanced assignments instead of unbalanced OT (ablation) | ASOT · Breakfast, YouTube Instructions, 50 Salads Mid/Eval, Desktop Assembly | MoF: 56.1 -> 29.7; 52.9 -> 39.4; 46.2 -> 39.7; 59.3 -> 35.7; 70.4 -> 56.5 | Mixed |
| Remove GW temporal consistency during training (ablation) | ASOT · Breakfast, YouTube Instructions, 50 Salads Mid/Eval, Desktop Assembly | MoF: 56.1 -> 34.4; 52.9 -> 41.1; 46.2 -> 29.0; 59.3 -> 35.1; 70.4 -> 49.5 | Hurt |
| Remove GW temporal consistency during training and testing (ablation) | ASOT · Breakfast, YouTube Instructions, 50 Salads Mid/Eval, Desktop Assembly | MoF: 56.1 -> 32.6; 52.9 -> 38.0; 46.2 -> 17.3; 59.3 -> 25.8; 70.4 -> 45.4 | Hurt |
Surprising findings
- Balanced OT slightly improves Desktop Assembly results, but severely degrades performance on Breakfast, YouTube Instructions and 50 Salads, reflecting long-tailed class distributions.
- Within-video per-video Hungarian matching gives ASOT weaker mIoU on Breakfast (18.6 full vs. 35.9 per video is not the same protocol; per-video mIoU underperforms), attributed to failing on short underrepresented actions.
Limitations
- Unsupervised learning pipeline is not effective on Breakfast with all activity categories; authors note a more sophisticated pipeline is needed.
- Underperforms on 50 Salads and mIoU on Breakfast under per-video evaluation, struggling with short underrepresented actions.
- Requires setting K to the ground-truth number of actions per activity category, and the GW radius and other hyperparameters are dataset-sensitive.
Experimental setup
Datasets: Breakfast YouTube Instructions 50 Salads Desktop Assembly
Benchmarks: Breakfast YouTube Instructions 50 Salads (Mid) 50 Salads (Eval) Desktop Assembly
Metrics: MoF F1-score mIoU segmental edit distance F1@10/25/50
Baselines: CTE VTE UDE ASAL TOT UFSA TW-FINCH ABD MS-TCN++
Builds on: TOT UFSA Gromov-Wasserstein optimal transport Fused Gromov-Wasserstein Sinkhorn/entropic OT
Closest papers by meaning
- Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · similarity 0.88)
- Hierarchical Vector Quantization for Unsupervised Action Segmentation (AAAI 2025 · similarity 0.88)
- Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment (CVPR 2024 · similarity 0.86)
- FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation (CVPR 2024 · similarity 0.86)
- ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation (NeurIPS 2024 · similarity 0.86)
- Minimizing Labeled, Maximizing Unlabeled: An Image-Driven Approach for Video Instance Segmentation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.