Research map / Label-efficient and robust learning / Knowledge distillation
Procedure Knowledge Decoupled Distillation Strategy for Procedure Planning in Instructional Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PKDD, a distillation strategy where a diffusion-transformer teacher sees real intermediate key frames and transfers action-level (binary BCE) and sequence-level (MSE) knowledge to weakly supervised procedure planners, improving success rate on CrossTask, COIN and NIV.
Problem
Weakly supervised non-autoregressive procedure planning in instructional videos lacks intermediate visual information, so models struggle to capture action semantics and relations between intermediate actions that are absent from start and goal observations.
Approach
A teacher diffusion transformer is given key intermediate frames (selected by cosine similarity to action labels), and its action distributions are decoupled into single-action binary cross-entropy distillation and sequence-level MSE distillation for a student planner, which can be plugged into existing weakly supervised models.
Key results
- On CrossTask with T=3, PKDD applied to PDPP raises success rate from 26.38 to 35.47.
- On COIN with T=4, PKDD on PDPP raises success rate from 14.38 to 20.71 and mAcc from 43.63 to 54.19.
- On NIV the gains are small and some metrics degrade, which the authors attribute to the small dataset causing overfitting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply PKDD to PDPP | PDPP · CrossTask T=3 | SR: 26.38 -> 35.47 | Helped |
| Apply PKDD to PDPP | PDPP · CrossTask T=4 | mAcc: 52.44 -> 59.20 | Helped |
| Apply PKDD to PDPP | PDPP · COIN T=3 | mAcc: 44.81 -> 55.90 | Helped |
| Apply PKDD to PDPP | PDPP · COIN T=4 | mIoU: 55.60 -> 66.14 | Helped |
| Apply PKDD to PDPP | PDPP · NIV T=3 | SR: 25.18 -> 27.04 | Helped |
| Apply PKDD to PDPP | PDPP · NIV T=4 | mIoU: 56.90 -> 55.91 | Hurt |
| Use SAKD without SDKD (ablation) | PDPP on CrossTask · CrossTask T=3 | SR: 35.47 -> 34.96 | Hurt |
| Use SDKD without SAKD (ablation) | PDPP on CrossTask · CrossTask T=3 | SR: 35.47 -> 35.23 | Hurt |
| Replace IIKD with direct intermediate action features (ablation) | PDPP on CrossTask · CrossTask T=3 | SR: 35.47 -> 33.74 | Hurt |
| Replace IIKD with standard KD (ablation) | PDPP on CrossTask · CrossTask T=3 | SR: 35.47 -> 35.06 | Hurt |
| Replace IIKD with KD using logit normalization (ablation) | PDPP on CrossTask · CrossTask T=3 | SR: 35.47 -> 35.15 | Hurt |
| Replace IIKD with DKD (ablation) | PDPP on CrossTask · CrossTask T=3 | SR: 35.47 -> 35.21 | Hurt |
Surprising findings
- On NIV, PKDD does not consistently improve results; some metrics drop (e.g., SCHEMA mAcc and SR at T=4), attributed to dataset size and overfitting.
Limitations
- Gains on NIV are limited and inconsistent, likely due to overfitting on a small dataset.
- Teacher requires access to real intermediate visual observations, which are unavailable at inference time.
- Gains vary across backbones; structural mismatch between teacher and student reduces the effect for SCHEMA and KEPP.
Experimental setup
Datasets: CrossTask COIN NIV HowTo100M
Benchmarks: CrossTask COIN NIV
Metrics: success rate mAcc mIoU
Baselines: KEPP SCHEMA PDPP DDN PlaTe Ext-GAIL KD DKD
Builds on: Hinton knowledge distillation PDPP KEPP SCHEMA DKD
Closest papers by meaning
- Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videos (CVPR 2024 · similarity 0.91)
- SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos (ICLR 2024 · similarity 0.90)
- Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos (ICLR 2025 · similarity 0.89)
- PlanLLM: Video Procedure Planning with Refinable Large Language Models (AAAI 2025 · similarity 0.87)
- Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations (NeurIPS 2025 · similarity 0.86)
- Generative Model-Based Feature Knowledge Distillation for Action Recognition (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.