Research map / Video / Action recognition
Progress-Aware Online Action Segmentation for Egocentric Procedural Task Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProTAS, an online action segmentation framework for egocentric procedural videos that makes offline backbones causal, adds an action progress prediction module and a learned task-graph loss, and improves online segmentation on GTEA, EgoProceL and EgoPER.
Problem
Online (streaming) fully-supervised action segmentation for egocentric procedural task videos, where offline-trained models degrade sharply under online inference and suffer from oversegmentation.
Approach
Make existing TCN and Transformer segmenters causal to remove the train/test mismatch, add a GRU-based per-action progress regressor whose outputs refine the frame predictions, and use a task graph (learned from training transcripts) as a penalty during training and inference to enforce procedure-consistent predictions.
Key results
- On EgoProceL with MSTCN, the full CAS+APP+TG model raises F1@0.5 from 8.8% (offline-trained model run online) to 35.7%.
- On EgoPER with ASFormer, Acc under online inference improves from 19.8% (offline-trained) to 71.7% (CAS+APP+TG).
- Adding APP consistently improves metrics across three datasets, e.g. about 4% F1@0.5 gain on EgoProceL and EgoPER with MSTCN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use causal convolutions or causal attention masking during training and inference | MSTCN and ASFormer · EgoProceL | accuracy: 13.2 to 64.8 | Helped |
| Add action progress prediction (ablation) | MSTCN CAS · EgoProceL | F1@0.5: 33.0 -> 35.2 | Helped |
| Add action progress prediction (ablation) | ASFormer CAS · EgoPER | F1@0.5: 44.7 -> 46.9 | Helped |
| Add transcript-based task graph inference (ablation) | MSTCN CAS+APP · EgoPER | Edit: 55.0 -> 60.7 | Helped |
| Add transcript-based task graph inference (ablation) | MSTCN CAS+APP · GTEA | accuracy: 76.0 -> 74.3 | Hurt |
| Add transcript-based task graph inference (ablation) | ASFormer CAS+APP · EgoProceL | F1@0.5: 36.1 -> 36.8 | Helped |
| Use the OODL online-offline discrepancy loss (ablation) | MSTCN CAS · GTEA | F1@0.5: 56.0 -> 54.1 | No clear effect |
| Replace transcript-based task graph with a learnable task graph (ablation) | ProTAS with MSTCN · GTEA | F1@0.5: 59.7 -> 60.2 | Helped |
| Replace transcript-based task graph with a learnable task graph (ablation) | ProTAS with MSTCN · EgoProceL | F1@0.5: 35.7 -> 34.8 | Hurt |
| Replace transcript-based task graph with a learnable task graph (ablation) | ProTAS with MSTCN · EgoPER | F1@0.5: 46.3 -> 47.1 | Helped |
| Use a manually designed rather than transcript-based task graph (ablation) | ProTAS with MSTCN · EgoPER | F1@0.5: 46.3 -> 46.5 | No clear effect |
| Replace the GRU progress regressor with ordinal regression (ablation) | APP · GTEA | F1@0.5 | Hurt |
Surprising findings
- Applying an offline-trained model directly online drops accuracy from 69.5% to 13.2% on EgoProceL with ASFormer.
- Adding an offline teacher via OODL does not help in fully-supervised online segmentation, since ground-truth frame labels already supervise training.
Limitations
- Gains are smaller on Assembly101 because grayscale egocentric views make progress hard to capture.
- Learnable task graph does not help on EgoProceL, attributed to large procedural variation.
- APP is ineffective for actions without clear appearance change progression (e.g., mixing).
Experimental setup
Datasets: GTEA EgoProceL EgoPER Assembly101
Benchmarks: GTEA EgoProceL EgoPER
Metrics: frame-wise accuracy edit score F1@{0.1,0.25,0.5} mean squared error E@C CMD
Baselines: MSTCN offline MSTCN online (offline-trained) CAS+OODL ASFormer offline ASFormer online
Builds on: MS-TCN ASFormer OODL Graph2Vid
Closest papers by meaning
- Error Detection in Egocentric Procedural Task Videos (CVPR 2024 · similarity 0.88)
- OnlineTAS: An Online Baseline for Temporal Action Segmentation (NeurIPS 2024 · similarity 0.88)
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation (NeurIPS 2025 · similarity 0.87)
- Understanding Multi-Task Activities from Single-Task Videos (CVPR 2025 · similarity 0.87)
- Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.87)
- Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.