Research map / Video / Action recognition
PREGO: Online Mistake Detection in PRocedural EGOcentric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PREGO, a one-class online mistake detector for procedural egocentric videos that compares MiniRoad-recognized current actions with LLM (zero-shot, in-context) predicted next actions, and introduces Assembly101-O and Epic-tent-O benchmarks.
Problem
Detecting open-set procedural mistakes online in egocentric videos, where no existing method handles unseen error types causally under one-class training.
Approach
Combine an online action recognizer with zero-shot LLM symbolic in-context next-action anticipation, flagging a mistake when recognized and anticipated actions diverge, trained only on correct procedures (one-class).
Key results
- On Assembly101-O, PREGO with LLaMA and MiniRoad recognition reaches 41.8 F1, while OadTR for MD reaches 20.7 F1 (about 102% improvement with the same recognizer on OadTR-based setup reported as 35.8 vs 20.7).
- PREGO with LLaMA and oracle recognition reaches 46.3 F1 on Assembly101-O, versus 31.8 for BERT, a 45.6% improvement.
- Random symbol prompts achieve the highest F1 (41.8) on Assembly101-O versus 41.4 for semantic and 39.9 for numerical prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PREGO with oracle recognition and LLaMA anticipation instead of BERT | Assembly101-O · Assembly101-O | F1 score: 31.8 -> 46.3 | Helped |
| Use PREGO with oracle recognition and LLaMA anticipation instead of BERT | Epic-tent-O · Epic-tent-O | F1 score: 10.4 -> 19.1 | Helped |
| Use symbolic LLaMA anticipation with OadTR recognition instead of OadTR mistake detection | Assembly101-O · Assembly101-O | F1 score: 20.7 -> 35.8 | Helped |
| Use symbolic LLaMA anticipation with OadTR recognition instead of OadTR mistake detection | Epic-tent-O · Epic-tent-O | F1 score: 10.2 -> 17.2 | Helped |
| Use LLaMA rather than GPT-3.5 for anticipation with oracle recognition | PREGO · Assembly101-O | F1 score: 42.1 -> 46.3 | Helped |
| Use LLaMA rather than GPT-3.5 for anticipation with oracle recognition | PREGO · Epic-tent-O | F1 score: 17.4 -> 19.1 | Helped |
| Represent actions with random symbols rather than numerical labels (ablation) | PREGO · Assembly101-O | F1 score: 39.9 -> 41.8 | Helped |
| Represent actions with random symbols rather than semantic labels (ablation) | PREGO · Assembly101-O | F1 score: 41.4 -> 41.8 | Helped |
| Use semantic labels rather than numerical labels for actions (ablation) | PREGO · Assembly101-O | F1 score: 39.9 -> 41.4 | Helped |
| Use referenced-context instructions rather than elaborate-context instructions (ablation) | PREGO · Assembly101-O | F1 score: 40.5 -> 41.8 | Helped |
| Use referenced-context instructions rather than unreferenced-context instructions (ablation) | PREGO · Assembly101-O | F1 score: 41.4 -> 41.8 | Helped |
| Use elaborate-context instructions rather than unreferenced-context instructions (ablation) | PREGO · Assembly101-O | F1 score: 41.4 -> 40.5 | Hurt |
Surprising findings
- Random symbol representations of actions in the prompt outperform semantic labels for next-step anticipation.
- Numerical action indices give the lowest F1, which the authors attribute to spurious ordering bias.
Limitations
- Small number of procedures (190 correct training procedures in Assembly101-O; 29 videos in Epic-tent-O) limits generalization.
- Low precision (roughly 8-30%) yields many false alarms.
- Pipeline is slow (0.02 fps), not real-time.
Experimental setup
Datasets: Assembly101 Epic-tent
Benchmarks: Assembly101-O Epic-tent-O
Metrics: precision recall F1 score
Baselines: One-step memory BERT OadTR for MD
Builds on: MiniRoad OadTR BERT
Closest papers by meaning
- Error Detection in Egocentric Procedural Task Videos (CVPR 2024 · similarity 0.91)
- Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos (NeurIPS 2024 · similarity 0.86)
- CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities (NeurIPS 2024 · similarity 0.86)
- Progress-Aware Online Action Segmentation for Egocentric Procedural Task Videos (CVPR 2024 · similarity 0.85)
- Modeling Multiple Normal Action Representations for Error Detection in Procedural Tasks (CVPR 2025 · similarity 0.84)
- Gazing Into Missteps: Leveraging Eye-Gaze for Unsupervised Mistake Detection in Egocentric Videos of Skilled Human Activities (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.