Research map / Large language models / Agents, code and math
SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCHEMA, which represents procedure steps as LLM-generated before/after state-change descriptions and aligns visual states with them via contrastive learning to plan action sequences in instructional videos.
Problem
Procedure planning in instructional videos under weak (step-sequence-only) supervision, where intermediate visual states are unannotated and the role of states in planning is underexplored.
Approach
Represent each step as before/after state-change language descriptions generated by GPT-3.5 with chain-of-thought prompting, align visual start/goal states with these descriptions via cross-modal contrastive learning, and use the descriptions as supervision for predicted mid-states, decomposing planning into mid-state prediction and step prediction.
Key results
- On CrossTask with T=3, SCHEMA reaches 31.83 SR, versus 26.40 for EGPP and 23.34 for P3IV.
- On COIN with T=3, SR is 32.09 versus 19.57 for EGPP; on NIV with T=3, SR is 27.93 versus 26.05 for EGPP.
- Ablation on CrossTask T=3: state alignment and mid-state prediction each add gains, with the full model giving the best SR (31.83 vs 28.72 baseline).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add state alignment (ablation) | SCHEMA · CrossTask T=3 | SR: 28.72 -> 30.15; 29.41 -> 31.83 | Helped |
| add mid-state prediction (ablation) | SCHEMA · CrossTask T=3 | SR: 28.72 -> 29.41; 30.15 -> 31.83 | Helped |
| use two decoders instead of one decoder (ablation) | SCHEMA · CrossTask T=3 | SR: 30.25 -> 31.83 | Helped |
| use two decoders instead of one decoder (ablation) | SCHEMA · CrossTask T=4 | SR: 19.01 -> 19.71 | Helped |
| use state-description external memory instead of randomly initialized memory (ablation) | SCHEMA · CrossTask T=3 | SR: 29.83 -> 31.83 | Helped |
| use state-description external memory instead of randomly initialized memory (ablation) | SCHEMA · CrossTask T=4 | SR: 19.53 -> 19.71 | Helped |
| use state descriptions rather than step descriptions as step-classifier memory (ablation) | SCHEMA · CrossTask T=3 | SR: 30.00 -> 31.83 | Helped |
| use state descriptions rather than step descriptions as step-classifier memory (ablation) | SCHEMA · CrossTask T=4 | SR: 19.30 -> 19.71 | Helped |
| apply Viterbi post-processing (ablation) | SCHEMA · CrossTask T=3 | SR: 27.48 -> 31.83 | Helped |
| apply Viterbi post-processing (ablation) | SCHEMA · CrossTask T=4 | SR: 14.85 -> 19.71 | Helped |
| use probabilistic rather than deterministic modeling (ablation) | SCHEMA · CrossTask T=3 | SR: 31.83 -> 29.51 | Hurt |
| use probabilistic rather than deterministic modeling (ablation) | SCHEMA · CrossTask T=4 | SR: 19.71 -> 16.55 | Hurt |
Surprising findings
- PDPP's advantage on CrossTask comes largely from a different start/end state window definition, which the authors argue makes comparisons unfair; under the same setting SCHEMA outperforms PDPP.
- The probabilistic variant underperforms the deterministic variant on SR, mAcc and mIoU, which the authors attribute to language supervision reducing visual uncertainty.
Limitations
- State changes not explicitly visible in video may be missed.
- Quality of state descriptions depends on LLM, with occasional hallucinations and merged steps.
- Restricted to a closed-vocabulary setting.
Experimental setup
Datasets: CrossTask COIN NIV HowTo100M
Benchmarks: CrossTask COIN NIV
Metrics: SR mAcc mIoU KL-Div NLL ModePrec ModeRec
Baselines: Random Retrieval-Based WLTDO UAAA UPN DDN PlaTe Ext-GAIL P3IV PDPP EGPP
Builds on: P3IV Zhao et al. probabilistic procedure planning (Viterbi post-processing) CLIP Socratic Models
Closest papers by meaning
- Procedure Knowledge Decoupled Distillation Strategy for Procedure Planning in Instructional Videos (AAAI 2025 · similarity 0.90)
- Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videos (CVPR 2024 · similarity 0.89)
- Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos (ICLR 2025 · similarity 0.87)
- PlanLLM: Video Procedure Planning with Refinable Large Language Models (AAAI 2025 · similarity 0.86)
- ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions (CVPR 2025 · similarity 0.85)
- Differentiable Learning of Lifted Action Schemas for Classical Planning (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.