Research map / Video / Action recognition
Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ATBA, which directly localizes action transitions via candidate boundary selection and a drop-allowed DP alignment (instead of frame-wise Viterbi/DTW) to generate pseudo labels for weakly-supervised action segmentation, plus video-level losses.
Problem
Weakly-supervised action segmentation from transcripts relies on slow, serial frame-by-frame alignment (Viterbi/DTW) to produce pseudo labels, and class-agnostic boundaries are noisy, so training is slow and accuracy suffers.
Approach
Treat pseudo-segmentation generation as detecting a few action transitions: generate more class-agnostic boundary candidates than transitions, score each candidate against each transition using class-specific pattern matching, and select the optimal subset with a drop-allowed DP alignment whose cost scales with transcript length, not video length. Video-level multi-label and global-local contrasti
Key results
- On Breakfast, ATBA reaches 53.9% MoF (±1.2), beating the previous best (MuCon 48.5 / DPDTW 50.8) by about 2.7 points.
- On CrossTask, ATBA reaches 50.6% MoF versus 42.8% for POC, a gain of +7.8% MoF.
- Training on Breakfast takes 3.45 hours versus 66.73 hours for CDFL, about 10x faster than frame-by-frame alignment methods on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ATBA instead of AdaAct | weakly-supervised action segmentation · Breakfast | MoF: 51.2 -> 53.9±1.2 | Helped |
| Use ATBA instead of DPDTW | weakly-supervised action segmentation · Hollywood Extended | MoF: 55.6 -> 47.7±2.5 | Hurt |
| Use ATBA instead of POC | weakly-supervised action segmentation · CrossTask | MoF: 42.8 -> 50.6±1.3 | Helped |
| Replace frame-by-frame alignment with ATBA | Breakfast training · Breakfast | training time: 11.23* -> 3.45 | Helped |
| Use only class-agnostic boundary scores to select transitions (ablation) | ATBA · Breakfast | MoF: 67.9 -> 38.6 | Hurt |
| Use only class-agnostic boundary scores to select transitions (ablation) | ATBA · Breakfast | pseudo-label accuracy: 62.3 -> 52.5 | Hurt |
| Add class-specific transition patterns to class-agnostic boundary selection (ablation) | ATBA · Breakfast | MoF: 38.6 -> 67.9 | Helped |
| Exclude class-agnostic boundary scores from transition alignment (ablation) | ATBA · Breakfast | MoF: 67.9 -> 51.7 | Hurt |
| Exclude class-agnostic boundary scores from transition alignment (ablation) | ATBA · Breakfast | pseudo-label accuracy: 62.3 -> 47.8 | Hurt |
| Add video-level multi-label supervision (ablation) | ATBA · Breakfast | MoF: 67.5 -> 66.6 | Helped |
| Add global-local contrastive loss (ablation) | ATBA with video-level multi-label loss · Breakfast | MoF: 66.6 -> 67.9 | Helped |
| Use separate classifiers for frame-wise and video-level prediction (ablation) | ATBA with video-level and global-local losses · Breakfast | MoF: 67.9 -> 68.1 | No clear effect |
Surprising findings
- Class-agnostic boundary detection alone (no transition scoring) gives only 38.6% MoF on Breakfast, showing that boundaries without class-specific transition patterns are very noisy.
Limitations
- On Hollywood Extended, gains are not significant because of many shot changes that create noisy boundaries.
- Relies on pseudo-label initialization via a two-stage training schedule using video-level labels first.
- Several hyperparameters (window sizes, mu, lambda, background sample weight) are set per dataset.
Experimental setup
Datasets: Breakfast Hollywood Extended CrossTask
Benchmarks: Breakfast Hollywood Extended CrossTask
Metrics: MoF MoF-Bg IoU IoD
Baselines: TCFPN+ISBA NN-Viterbi CDFL DPDTW TASL MuCon POC AdaAct D3TW
Builds on: UBoCo Viterbi alignment dynamic time warping POC MuCon
Closest papers by meaning
- Fast and Unsupervised Action Boundary Detection for Action Segmentation (CVPR 2022 · similarity 0.90)
- ASM-Loc: Action-Aware Segment Modeling for Weakly-Supervised Temporal Action Localization (CVPR 2022 · similarity 0.90)
- FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation (CVPR 2024 · similarity 0.90)
- Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · similarity 0.89)
- Realigning Confidence with Temporal Saliency Information for Point-Level Weakly-Supervised Temporal Action Localization (CVPR 2024 · similarity 0.88)
- Efficient Temporal Action Segmentation via Boundary-aware Query Voting (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.