Research map / Detection and segmentation / CNN features
Efficient Temporal Action Segmentation via Boundary-aware Query Voting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BaFormer, a single-stage query-based Transformer that predicts instance-query class/mask pairs plus a global boundary query, then classifies boundary-delimited segments by query voting for efficient temporal action segmentation.
Problem
Temporal action segmentation methods achieve high accuracy but at large computational cost due to dense frame-wise processing, multi-stage refinement, and time-consuming post-processing such as Viterbi decoding.
Approach
Treat each video segment as an instance token via mask classification (MaskFormer-style queries with soft mask attention), add a global query for class-agnostic boundary prediction to form continuous segment proposals, and assign each proposal a class by majority-weighted query voting instead of frame-wise or Viterbi post-processing.
Key results
- On 50Salads, BaFormer reaches 89.3% accuracy, 88.4 F1@25 and 84.2 Edit, versus 90.1 accuracy-level DiffAct at 43.94 GFLOPs and 2.306 s; BaFormer runs in 0.139 s (about 6% of DiffAct's time) at 4.54 GFLOPs.
- Compared with the single-stage UVAST with alignment decoder, BaFormer uses 24% of its running time while improving accuracy by 10% on 50Salads.
- Query-based voting beats frame-based voting by 1.8% accuracy, 5.6 Edit and 6.7 F1@50 on 50Salads.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase Transformer decoder depth from 8 to 10 layers (ablation) | BaFormer · 50Salads | accuracy: 88.4 -> 89.5 | Helped |
| reduce Transformer decoder depth from 10 to 3 layers (ablation) | BaFormer · 50Salads | accuracy: 89.5 -> 84.2 | Hurt |
| increase query count from 50 to 70 (ablation) | BaFormer · 50Salads | accuracy: 87.1 -> 87.9 | Helped |
| increase query count from 100 to 120 (ablation) | BaFormer · 50Salads | accuracy: 89.5 -> 87.0 | Hurt |
| increase query count from 100 to 150 (ablation) | BaFormer · 50Salads | accuracy: 89.5 -> 84.4 | Hurt |
| add multi-level frame-decoder features instead of single-level features (ablation) | BaFormer · 50Salads | accuracy: +5.5% | Helped |
| add auxiliary losses with multi-level features (ablation) | BaFormer · 50Salads | accuracy: 87.4 -> 89.5 | Helped |
| add auxiliary losses with single-level features (ablation) | BaFormer · 50Salads | accuracy: 83.5 -> 84.0 | Helped |
| use instance matching instead of ordered class matching with the minimum feasible query count (ablation) | BaFormer · 50Salads | accuracy: -1.8% | Hurt |
| increase instance-matching queries from 26 to 100 (ablation) | BaFormer · 50Salads | accuracy | Helped |
| use transcript matching instead of instance matching (ablation) | BaFormer · 50Salads | evaluation metrics | Hurt |
| use an average-pooled global query instead of a class token (ablation) | BaFormer · 50Salads | predictive accuracy | Helped |
Surprising findings
- Using the average-pooled instance query as the global boundary query outperforms a learned class token, which the authors attribute to intermediate query-based supervision.
- Increasing queries beyond 100 (to 120 and 150) reduces performance on 50Salads, which the authors attribute to overfitting.
Limitations
- Boundary prediction quality is a bottleneck; ground-truth boundaries substantially improve results, per the authors' analysis.
- Performance is sensitive to query count and decoder depth, requiring tuning on 50Salads.
- Efficiency is reported on 50Salads only; running time and FLOPs on GTEA and Breakfast are omitted.
Experimental setup
Datasets: GTEA 50Salads Breakfast
Benchmarks: GTEA 50Salads Breakfast
Metrics: accuracy F1@10 F1@25 F1@50 Edit score running time FLOPs
Baselines: MSTCN SSTDA BCN HASR DTGRM ASRF ASFormer UARL DTL RTK LtContext DiffAct
Builds on: MaskFormer Mask2Former ASFormer UVAST DETR
Closest papers by meaning
- Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment (CVPR 2024 · similarity 0.88)
- Fast and Unsupervised Action Boundary Detection for Action Segmentation (CVPR 2022 · similarity 0.87)
- SCT: Set Constrained Temporal Transformer for Set Supervised Action Segmentation (CVPR 2020 · similarity 0.87)
- Fast Learning of Temporal Action Proposal via Dense Boundary Generator (AAAI 2020 · similarity 0.87)
- OnlineTAS: An Online Baseline for Temporal Action Segmentation (NeurIPS 2024 · similarity 0.86)
- Condensing Action Segmentation Datasets via Generative Network Inversion (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.