Research map / Video / Action recognition
OnlineTAS: An Online Baseline for Temporal Action Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OnlineTAS, the first online framework for temporal action segmentation, combining a GRU-plus-adaptive-memory context-aware feature augmentation module with a confidence- and length-based post-processing step to curb over-segmentation.
Problem
Temporal action segmentation is studied almost exclusively offline; online segmentation, which must rely only on past and current frames, suffers from severe over-segmentation and under-used temporal context.
Approach
A context-aware feature augmentation module attends over a per-video adaptive memory (short-term clip memory plus long-term memory tokens with a two-thirds cap and FIFO eviction) on top of a GRU, feeding a causal single-stage TCN; a post-processing step relabels low-confidence frames that would create short segments.
Key results
- On 50Salads, the semi-online model with post-processing reaches 75.0 Edit and 68.0 F1@50, versus 19.6 Edit for the causal TCN baseline in online mode.
- On Breakfast, online inference with post-processing raises F1@50 from 9.3 to 30.5 over the same configuration without post-processing, with Acc dropping from 56.7 to 52.9.
- The method reports 33.8 FPS for the full raw-RGB pipeline and 238.1 FPS with pre-computed I3D features on an Nvidia A40.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CFA to the causal TCN | Causal TCN · 50Salads | Acc: 75.2% vs. 80.9% | Helped |
| Add post-processing | OnlineTAS · Breakfast, online | F1@50: 9.3% to 30.5% | Helped |
| Use semi-online rather than online inference | OnlineTAS · Across the three datasets | Segmentation metrics | Helped |
| Add GRU (ablation) | Causal TCN · 50Salads | Edit: 19.6 to 27.1 | Helped |
| Add CFA without context memory (ablation) | Causal TCN · 50Salads | Acc: 75.2 to 76.2 | Helped |
| Add GRU and CFA (ablation) | Causal TCN · 50Salads | Acc: 75.2 to 79.1 | Helped |
| Add adaptive memory and CFA without GRU (ablation) | Causal TCN · 50Salads | Acc: 75.2 to 78.9 | Helped |
| Add GRU to CFA with adaptive memory (ablation) | OnlineTAS · 50Salads | Acc: 78.9 to 82.4 | Helped |
| Vary CFA interaction iterations from 2 (ablation) | OnlineTAS · 50Salads | Acc: 79.1; 79.5; 79.1; 79.2 | No clear effect |
| Combine short- and long-term memory (ablation) | OnlineTAS · 50Salads | Acc: 80.3; 80.4; 82.4 | Helped |
| Increase clip and memory size to 256 (ablation) | OnlineTAS · 50Salads | Seg.: 37.9 to 38.2 | Helped |
Surprising findings
- Adapting the online TAD model LSTR directly to segmentation yields very poor segmental scores (e.g. 5.0 Edit on 50Salads), even below the causal TCN baseline.
- Semi-online inference (non-overlapping clips) outperforms true frame-by-frame online inference across metrics, attributed to preserved local temporal continuity.
Limitations
- Evaluated only on cooking videos, not diverse real-world videos.
- Interrupted actions and very long streams are challenging given the fixed memory budget.
- Persistent over-segmentation occurs when the network makes incorrect but confident predictions.
Experimental setup
Datasets: Breakfast 50Salads GTEA
Benchmarks: Breakfast 50Salads GTEA
Metrics: frame-wise accuracy segmental edit score segmental F1@{10,25,50}
Baselines: Causal TCN LSTR MV-TAS MS-TCN ASFormer DiffAct
Builds on: MS-TCN LSTR Swin Transformer Transformer decoder
Closest papers by meaning
- Temporal Action Segmentation From Timestamp Supervision (CVPR 2021 · similarity 0.90)
- Temporal Action Detection Using a Statistical Language Model (CVPR 2016 · similarity 0.89)
- FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation (CVPR 2024 · similarity 0.88)
- Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · similarity 0.88)
- Progress-Aware Online Action Segmentation for Egocentric Procedural Task Videos (CVPR 2024 · similarity 0.88)
- ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.