Research map / Label-efficient and robust learning / OOD and anomaly detection
Modeling Multiple Normal Action Representations for Error Detection in Procedural Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AMNAR, which predicts all valid next actions from a task graph via dynamic programming, reconstructs a normal action representation for each, and flags an error if the ongoing action is far from all of them.
Problem
Error detection in procedural activities, where static prototypes fail under distribution shift and multiple distinct next actions can be valid after the same executed sequence.
Approach
Dynamically generate multiple normal action representations conditioned on executed actions, one per valid next action predicted from a task graph with DP-based sequence filtering, and match the ongoing action to the closest one with a per-class threshold.
Key results
- On EgoPER, AMNAR reaches 64.4% average EDA and 68.5% AUC, improving over EgoPED by 7.4% EDA and 6.5% AUC.
- On HoloAssist, AMNAR achieves 69.9% EDA and 56.5% AUC, versus 66.2% and 55.2% for EgoPED.
- On CaptainCook4D, AMNAR gets 72.3% EDA and 60.2% AUC, versus 69.8% and 54.9% for EgoPED.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AMNAR instead of EgoPED | EgoPER · EgoPER (All) | EDA: +7.4% | Helped |
| use AMNAR instead of EgoPED | EgoPER · EgoPER (All) | AUC: +6.5% | Helped |
| use AMNAR instead of EgoPED | HoloAssist · HoloAssist (All) | EDA: +3.7% | Helped |
| use AMNAR instead of EgoPED | HoloAssist · HoloAssist (All) | AUC: +1.3% | Helped |
| use AMNAR instead of EgoPED | CaptainCook4D · CaptainCook4D | EDA: +2.5% | Helped |
| use AMNAR instead of EgoPED | CaptainCook4D · CaptainCook4D | AUC: +5.3% | Helped |
| add PAPB contextual valid-action prediction (ablation) | AMNAR · EgoPER (All) | EDA: +5.8% | Helped |
| add PAPB contextual valid-action prediction (ablation) | AMNAR · EgoPER (All) | AUC: +5.5% | Helped |
| add RRB context-aware normal representation reconstruction (ablation) | AMNAR · EgoPER (All) | EDA: +4.4% | Helped |
| add RRB context-aware normal representation reconstruction (ablation) | AMNAR · EgoPER (All) | AUC: +4.5% | Helped |
| use cluster-centered rather than text-based action representations (ablation) | AMNAR · EgoPER (All) | AUC: 64.1 -> 68.5 | Helped |
| predict cluster-center residuals rather than full features directly (ablation) | AMNAR · EgoPER (All) | AUC: +6.3% | Helped |
Surprising findings
- Using DINOv2 features instead of I3D raises EDA to 71.0% on EgoPER, although AUC drops to 69.2% relative to the default setting's 68.5% AUC being higher on the other metric.
- Even when EgoPED uses AMNAR's action segmentation results, it still underperforms AMNAR in error detection, indicating gains are not solely from segmentation.
Limitations
- Relies on a predefined or training-set-constructed task graph to determine valid next actions.
- Performance drops modestly when previous errors are present in the action sequence (EDA 64.3% vs 70.8%).
Experimental setup
Datasets: EgoPER HoloAssist CaptainCook4D
Benchmarks: EgoPER HoloAssist CaptainCook4D
Metrics: EDA AUC Precision
Baselines: HF2-VAD HF2-VAD + SSPCAB S3R EgoPED Random
Builds on: EgoPED ActionFormer I3D
Closest papers by meaning
- Error Detection in Egocentric Procedural Task Videos (CVPR 2024 · similarity 0.86)
- CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities (NeurIPS 2024 · similarity 0.85)
- SAFE: Multitask Failure Detection for Vision-Language-Action Models (NeurIPS 2025 · similarity 0.85)
- AED: Adaptable Error Detection for Few-shot Imitation Policy (NeurIPS 2024 · similarity 0.85)
- PREGO: Online Mistake Detection in PRocedural EGOcentric Videos (CVPR 2024 · similarity 0.84)
- ATI-VLA: Action-Centric Predictive Vision–Language–Action Models via Actionable Alignment Then Adaptive Injection (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.