Research map / Detection and segmentation / Open-set and few-shot detection
Error Detection in Egocentric Procedural Task Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the EgoPER egocentric procedural error dataset and EgoPED, a framework that detects procedural errors by training only on error-free videos, combining action segmentation, active-object GCN features, and contrastive step prototype learning.
Problem
Detecting procedural errors (omission, addition, modification, slip) in egocentric task videos when only error-free videos are available for training, a setting poorly served by existing video anomaly detection methods built for static surveillance views.
Approach
Combine holistic features from a temporal action segmentation backbone with relational features from a GCN over active-object graphs, and learn multiple per-step prototypes via contrastive learning; errors are flagged when frames are dissimilar to all normal prototypes of their predicted step.
Key results
- On EgoPER, EgoPED (AF) reaches 57.0% EDA averaged over tasks versus 43.0% for S3R and 27.1% for HF2-VAD.
- On HoloAssist, EgoPED improves EDA by about 17% over baselines for both verb and noun labels (71.0% noun EDA with AF vs 51.6% for S3R).
- Active object detection with GCN improves EgoPER EDA by 4.5% and AUC by 3.3%; using 2 prototypes per step improves EDA by 1.5% and AUC by 3.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EgoPED with ActionFormer instead of the strongest reported baseline | EgoPER · EgoPER (all tasks) | EDA: 43.0 -> 57.0 | Helped |
| use EgoPED with ActionFormer instead of the strongest reported baseline | EgoPER · EgoPER (all tasks) | AUC: 59.9 -> 62.0 | Helped |
| use EgoPED with ActionFormer instead of S3R | HoloAssist verb labels · HoloAssist (verb labels) | EDA: 51.2 -> 68.0 | Helped |
| use EgoPED with ActionFormer instead of S3R | HoloAssist noun labels · HoloAssist (noun labels) | EDA: 51.6 -> 71.0 | Helped |
| use EgoPED with DiffACT instead of ATA | ATA · ATA | Edit: 68.8 -> 74.0 | Helped |
| use 1 prototype per step instead of 2 (ablation) | EgoPED (AF) · EgoPER (all tasks) | EDA: 57.0 -> 55.5 | Hurt |
| use 1 prototype per step instead of 2 (ablation) | EgoPED (AF) · EgoPER (all tasks) | AUC: 62.0 -> 58.2 | Hurt |
| use 3 prototypes per step instead of 2 (ablation) | EgoPED (AF) · EgoPER (all tasks) | EDA: 57.0 -> 55.4 | Hurt |
| use 3 prototypes per step instead of 2 (ablation) | EgoPED (AF) · EgoPER (all tasks) | AUC: 62.0 -> 58.8 | Hurt |
| use 4 prototypes per step instead of 2 (ablation) | EgoPED (AF) · EgoPER (all tasks) | EDA: 57.0 -> 55.7 | Hurt |
| use 4 prototypes per step instead of 2 (ablation) | EgoPED (AF) · EgoPER (all tasks) | AUC: 62.0 -> 57.8 | Hurt |
| use 1 negative segment instead of 2 (ablation) | EgoPED (AF) · EgoPER (all tasks) | EDA: 57.0 -> 57.4 | Mixed |
Surprising findings
- Better action segmentation accuracy does not necessarily translate to better error detection: MSTCN++ has higher F1@0.5 than AF but lower error-detection AUC.
- Using training transcripts for omission detection performs close to using the ground-truth task graph.
Limitations
- Offline setting only: the method needs the full video at inference time and is not evaluated for online error detection.
- Omission errors often co-occur with other error types, making omission detection challenging.
- Number of prototypes must be set manually; automatic selection is left for future work.
Experimental setup
Datasets: EgoPER HoloAssist ATA
Benchmarks: EgoPER HoloAssist ATA
Metrics: Error Detection Accuracy (EDA) AUC Omission Accuracy (O-Acc) Omission IoU (O-IoU) Acc IoU Edit F1@0.5 F1error
Baselines: HF2-VAD HF2-VAD + SSPCAB S3R Random MuCon TASL CDFL ATA
Builds on: ActionFormer MSTCN++ DiffAct InfoNCE I3D
Closest papers by meaning
- PREGO: Online Mistake Detection in PRocedural EGOcentric Videos (CVPR 2024 · similarity 0.91)
- Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos (NeurIPS 2024 · similarity 0.90)
- Progress-Aware Online Action Segmentation for Egocentric Procedural Task Videos (CVPR 2024 · similarity 0.88)
- CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities (NeurIPS 2024 · similarity 0.87)
- Modeling Multiple Normal Action Representations for Error Detection in Procedural Tasks (CVPR 2025 · similarity 0.86)
- EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.