Research map / Video / Action recognition
CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CaptainCook4D, an egocentric 4D (RGB, depth, audio, IMU, tracking) dataset of 384 cooking recordings with intentional and unintentional errors, and benchmarks error recognition, multi-step localization, and procedure learning.
Problem
Existing procedural activity datasets contain only error-free executions, and error datasets focus on assembly with constant-shape objects, so AI systems cannot learn to recognize errors in procedures where objects continuously change shape and color, such as cooking.
Approach
An egocentric, multimodal dataset of real-kitchen cooking recipes performed both by following task-graph topological orders and by deliberately or unintentionally deviating, with step-level error labels and error categories, plus baselines for supervised and zero-shot error recognition, robust multi-step localization, and procedure learning.
Key results
- Zero-shot error recognition with VLMs is weak: Video-LLaVa V2 reaches 41.8 F1 and TimeChat V2 reaches 46.1 F1 on error recognition.
- Omnivore-based ActionFormer MSL achieves up to 48.16 mAP at IoU 0.1 on split P.
- Robust MSL models trained only on normal recordings drop sharply on error recordings (e.g., Omnivore split E: 28.59 mAP on normal vs 9.94 on error at IoU 0.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use error-category-specific prompt ensembling instead of a single step prompt (ablation) | Video-LLaVa · ZeroShotER | F1: 6.7 -> 41.8 | Helped |
| Use error-category-specific prompt ensembling instead of a single step prompt (ablation) | TimeChat · ZeroShotER | F1: 2.26 -> 46.1 | Helped |
| Use error-category-specific prompt ensembling instead of a single step prompt (ablation) | Video-LLaVa · ZeroShotER | accuracy: 64.3 -> 52.85 | Hurt |
| Use error-category-specific prompt ensembling instead of a single step prompt (ablation) | TimeChat · ZeroShotER | accuracy: 65.0 -> 43.5 | Hurt |
| Use Omnivore features instead of other tested video features | MSL models · CaptainCook4D MSL | mAP | Helped |
| Use video features extracted from longer than 1-second sub-segments | MSL models · CaptainCook4D MSL | performance | Helped |
| Evaluate normal-only-trained MSL models on error recordings rather than normal recordings | RobustMSL models · RobustMSL | mAP at IoU 0.1: 34.65 -> 12.51 | Hurt |
| Train the procedure-learning embedder with Cycleback Regression rather than the random setting | Procedure Learning · Average of 24 recipes | IOU: 2.22 -> 5.78 | Helped |
| Blend Cycleback Regression with Contrastive Inverse Difference Moment rather than the random setting | Procedure Learning · Average of 24 recipes | IOU: 2.22 -> 5.82 | Helped |
| Blend Cycleback Regression with Contrastive Inverse Difference Moment rather than using Cycleback Regression alone (ablation) | Procedure Learning · Average of 24 recipes | precision: 15.62 -> 15.78 | Helped |
| Blend Cycleback Regression with Contrastive Inverse Difference Moment rather than using Cycleback Regression alone (ablation) | Procedure Learning · Average of 24 recipes | recall: 10.85 -> 10.68 | Hurt |
| Blend Cycleback Regression with Contrastive Inverse Difference Moment rather than using Cycleback Regression alone (ablation) | Procedure Learning · Average of 24 recipes | IOU: 5.78 -> 5.82 | Helped |
Surprising findings
- Models trained only on normal recordings perform much worse on error recordings, showing that technique errors degrade multi-step localization robustness.
- Video-LLaVa and TimeChat with single-prompt zero-shot error recognition show very low recall (3.9 and 1.15), indicating they largely fail to flag errors.
Limitations
- Errors captured form only a subset of the combinatorial space of possible errors.
- Data collected in real kitchens within a limited geographic area with eight participants.
- Many induced errors are unintentional, and error labels depend on participant self-reporting.
Experimental setup
Datasets: CaptainCook4D
Benchmarks: CaptainCook4D
Metrics: accuracy precision recall F1 AUC mAP R@1 R@5 IoU
Baselines: Video-LLaVa TimeChat ActionFormer-based MSL models (3D ResNet, SlowFast, VideoMAE, Omnivore) EgoProceL embedder losses (Cycleback Regression, Contrastive-IDM) Random
Builds on: ActionFormer EgoProceL Omnivore ImageBind Whisper
Closest papers by meaning
- Error Detection in Egocentric Procedural Task Videos (CVPR 2024 · similarity 0.87)
- PREGO: Online Mistake Detection in PRocedural EGOcentric Videos (CVPR 2024 · similarity 0.86)
- Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos (NeurIPS 2024 · similarity 0.85)
- Modeling Multiple Normal Action Representations for Error Detection in Procedural Tasks (CVPR 2025 · similarity 0.85)
- IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants (NeurIPS 2025 · similarity 0.83)
- EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.