Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KEPP, which decomposes instructional-video procedure planning into predicting start/end steps with a conditioned projected diffusion step model and then generating the full plan with a diffusion planner conditioned on a probabilistic procedural knowledge graph (P2KG) built from training plans.
Problem
Procedure planning in instructional videos: generating a sequence of action steps from start and goal visual observations, where existing methods rely on heavy annotations (intermediate visuals, task labels, step text) and struggle with step-ordering causal constraints and multiple viable plans.
Approach
Decompose planning into predicting the first and last steps from visual states, then retrieve the most probable graph path between them from a probabilistic procedural knowledge graph built only from training procedure plans, and use it as a condition for a diffusion planner, requiring only minimal supervision.
Key results
- On CrossTask (T=3, conventional setting), KEPP with P2KG (R=1) reaches 33.34% SR versus 28.85% for SkipPlan.
- Under PDPP's setting on CrossTask T=4, KEPP reaches 24.15% SR versus 21.48% for PDPP.
- On COIN the method does not rank top at T=3 or T=4 (e.g., 20.25% SR vs 23.65% for SkipPlan at T=3), attributed to short average action sequences.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add P2KG plan recommendations (ablation) | KEPP · CrossTask T=3 (PDPP setting) | SR: 35.69 -> 38.12 | Helped |
| Add P2KG plan recommendations (ablation) | KEPP · CrossTask T=4 (PDPP setting) | SR: 20.52 -> 24.15 | Helped |
| Add P2KG plan recommendations (ablation) | KEPP · CrossTask T=6 (conventional) | SR: 8.16 -> 9.23 | Helped |
| Replace frequency-based graph weights with probabilistic out-edge-normalized weights (ablation) | KEPP · CrossTask T=6 | SR: 7.66 -> 8.09 | Helped |
| Train with ground-truth step data augmentation (ablation) | KEPP · CrossTask T=3 (PDPP setting) | SR: 38.12 -> 32.45 | Hurt |
| Use an LLM-generated plan recommendation instead of P2KG (ablation) | KEPP · CrossTask T=6 (PDPP setting, Llama-2-13B) | SR: 9.27 -> 7.74 | Hurt |
| Use an LLM-generated plan recommendation instead of P2KG (ablation) | KEPP · CrossTask T=6 (PDPP setting, Llama-2-70B) | SR: 9.27 -> 8.62 | Hurt |
| Use two P2KG recommendations instead of one (ablation) | KEPP · CrossTask T=3 (conventional) | SR: 33.34 -> 33.38 | No clear effect |
| Use two P2KG recommendations instead of one (ablation) | KEPP · CrossTask T=4 (conventional) | SR: 20.38 -> 21.02 | Helped |
| Use two P2KG recommendations instead of one (ablation) | KEPP · CrossTask T=6 (conventional) | SR: 8.09 -> 9.23 | Helped |
Surprising findings
- Using ground-truth start/end steps as augmented training data for the planner hurt test performance (SR 38.12 to 32.45 at T=3).
- On COIN, the method does not rank top at T=3 or T=4, which the authors attribute to short average action sequences.
Limitations
- P2KG recommendations are limited to steps seen in training data, so generalization to unseen procedural activities is weaker than LLM-based recommendations.
- Success rate drops sharply with longer horizons (about 40% at T=3 to 10% at T=6).
- Step model start/end prediction accuracy is modest (about 50-57% on CrossTask for non-feature-based setting), limiting planning performance.
Experimental setup
Datasets: CrossTask COIN Narrated Instructional Videos (NIV)
Benchmarks: CrossTask COIN NIV
Metrics: success rate mAcc mIoU
Baselines: WLTDO UAAA UPN DDN PlaTe Ext-GAIL P3IV PDPP SkipPlan E3P
Builds on: PDPP (conditioned projected diffusion) SkipPlan DDPM Probabilistic Procedural Knowledge Graph PKG
Closest papers by meaning
- Procedure Knowledge Decoupled Distillation Strategy for Procedure Planning in Instructional Videos (AAAI 2025 · similarity 0.91)
- SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos (ICLR 2024 · similarity 0.89)
- Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos (ICLR 2025 · similarity 0.85)
- VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting (AAAI 2025 · similarity 0.85)
- ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions (CVPR 2025 · similarity 0.85)
- VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.