Research map / Video / Action recognition
Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns task graphs from key-step sequences by directly maximizing a differentiable likelihood (TGML loss), via a direct-optimization adjacency matrix and a feature-based transformer (TGT), and applies the learned graphs to online mistake detection.
Problem
Task graphs encoding partial orderings of key-steps in procedural activities are usually hand-crafted from videos or text; the paper targets learning them end-to-end from action sequences and using them for online mistake detection in egocentric videos.
Approach
Frames task graph learning as maximum likelihood over sequences of key-steps, yielding a differentiable TGML loss that directly optimizes the edge weights of an adjacency matrix (DO) or predicts it from key-step embeddings with a transformer (TGT); the learned graph flags a mistake when an action's predicted pre-conditions are unmet.
Key results
- DO improves graph generation F1 by roughly +15.5 to +18.1 over the best competitor MSG^2 on CaptainCook4D; the abstract reports up to +16.7%.
- On Assembly101-O with ground-truth actions, DO reaches 75.9 average F1 vs 56.1 for MSG^2 (+19.8).
- On EPIC-Tent-O with ground-truth actions, TGT-text reaches 64.1 average F1 vs 56.6 for Count-Based, and DO gives +7.5 over the best competitor.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Direct optimization of adjacency matrix with TGML loss (DO) vs prior graph generation methods | task graph generation on CaptainCook4D · CaptainCook4D graph generation | F1: +15.5 to +18.1 over MSG^2 | Helped |
| Feature-based TGT instead of DO for graph generation | task graph generation on CaptainCook4D · CaptainCook4D graph generation | F1 | Hurt |
| Using DO-learned task graph for mistake detection instead of baselines | online mistake detection on Assembly101-O with ground truth actions · Assembly101-O | Avg F1: +19.8 | Helped |
| Using TGT-text task graph for mistake detection on EPIC-Tent-O | online mistake detection on EPIC-Tent-O with ground truth actions · EPIC-Tent-O | Avg F1: +7.5 (TGT-text 64.1 vs 56.6) | Helped |
| Using DO task graph for mistake detection on predicted action sequences | online mistake detection on Assembly101-O with MiniRoad predicted actions · Assembly101-O (predicted) | Avg F1: +7.3 | Helped |
| Using DO/TGT task graphs for mistake detection on predicted actions in EPIC-Tent-O | online mistake detection on EPIC-Tent-O with predicted actions · EPIC-Tent-O (predicted) | Avg F1: +1.3 | Helped |
| Single unified TGT-text model across all procedures vs per-procedure training (ablation) | TGT-text on CaptainCook4D · CaptainCook4D graph generation | F1: 80.8 to 64.5 | Hurt |
| Learned graph-based mistake detection with predicted graphs (even suboptimal) vs PREGO implicit forecasting | online mistake detection · Assembly101-O and EPIC-Tent-O | Avg F1 | Helped |
| Count-Based graph vs TGT-text graph for mistake detection on predicted actions (mistake F1) | online mistake detection on Assembly101-O (predicted) · Assembly101-O (predicted) | Mistake F1: -5.7 | Hurt |
| Mistake detection threshold/perturbation noise on ground-truth actions (20% perturbation) (ablation) | online mistake detection · Assembly101-O and EPIC-Tent-O | Avg F1: drop of 10-15 points at 20% perturbation | Hurt |
| Training TGT on video embeddings for pairwise ordering and future prediction | TGT (video embeddings, EgoVLPv2) · CaptainCook4D pairwise ordering and future prediction | accuracy | Helped |
Surprising findings
- The feature-based TGT model, trained only on key-step video embeddings, shows emerging video understanding (pairwise ordering and future prediction) above random chance without being explicitly trained for those tasks.
- The simple Count-Based baseline obtains a high correct-segment precision/recall imbalance, achieving only 4.8 precision on Assembly101-O correct segments.
Limitations
- Sensitive to noisy action sequences and tends to hallucinate pre-conditions due to limited sequence diversity.
- Does not explicitly model optional or repeatable key-steps.
- Requires labeled key-step sequences; cannot learn from unlabeled video or text alone.
Experimental setup
Datasets: CaptainCook4D Assembly101-O EPIC-Tent-O Ego-Exo4D
Benchmarks: CaptainCook4D Assembly101-O EPIC-Tent-O
Metrics: precision recall F1 accuracy
Baselines: MSGI MSG^2 Count-Based LLM (ChatGPT) PREGO
Builds on: PREGO MiniRoad EgoVLPv2
Closest papers by meaning
- Error Detection in Egocentric Procedural Task Videos (CVPR 2024 · similarity 0.90)
- PHGC: Procedural Heterogeneous Graph Completion for Natural Language Task Verification in Egocentric Videos (CVPR 2025 · similarity 0.87)
- PREGO: Online Mistake Detection in PRocedural EGOcentric Videos (CVPR 2024 · similarity 0.86)
- Progress-Aware Online Action Segmentation for Egocentric Procedural Task Videos (CVPR 2024 · similarity 0.86)
- CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities (NeurIPS 2024 · similarity 0.85)
- Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.