Research map / Video / Action recognition
Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MTID, a diffusion model for procedure planning in instructional videos that uses a learnable latent-space temporal interpolation module for mid-state visual supervision, plus action masking and a task-adaptive masked proximity loss.
Problem
Procedure planning in instructional videos: generating task-aligned action sequences from start and end visual observations, where intermediate states are unobserved and text-level supervision captures temporal relations poorly.
Approach
Generate intermediate latent visual features via a learnable interpolation matrix between start and end observation latents, inject them into a U-Net diffusion denoiser through cross-attention for end-to-end training, restrict action generation with a task-conditioned action mask, and weight the loss toward endpoints with a mask-scaled proximity term.
Key results
- On CrossTask with T=3, MTID reaches 40.45 SR, 67.19 mAcc and 69.17 mIoU, versus 38.93 SR for SCHEMA and 37.2 SR for PDPP.
- On COIN (KEPP setting, T=4), MTID reaches 22.74 SR and 49.90 mAcc, above SCHEMA's 22.02 SR and 45.33 mAcc.
- Ablation on CrossTask (T=3) shows the interpolation module, masked projection and proximity loss each add gains over PDPP (37.20 to 40.45 SR combined).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MTID temporal interpolation and masked diffusion to the procedure planner | CrossTask · CrossTask T=3 | SR: 38.93 -> 40.45 | Helped |
| Add MTID temporal interpolation and masked diffusion to the procedure planner | COIN · COIN T=3 | mAcc: 49.84 -> 51.70 | Helped |
| Add MTID temporal interpolation and masked diffusion to the procedure planner | NIV · NIV T=3 | SR: 27.93 -> 28.52 | Helped |
| Add gradient weighted loss and task-adaptive mask to MSE (ablation) | MTID · CrossTask T=5 | SR: 11.89 -> 15.26 | Helped |
| Apply masked projection at initialization rather than removing it (ablation) | MTID · CrossTask T=3 | SR: 39.17 -> 40.45 | Helped |
| Apply masked projection during denoising iterations (ablation) | MTID · CrossTask T=3 | SR: 40.45 -> 3.38 | Hurt |
| Add the observation encoder and transformer blocks to interpolation (ablation) | MTID · CrossTask T=3 | SR: 37.86 -> 40.45 | Helped |
| Increase model size from 256 to 512 (ablation) | MTID · CrossTask T=3 | SR: 40.45 -> 37.94 | Hurt |
| Increase model size from 256 to 512 (ablation) | MTID · COIN T=3 | SR: 28.84 -> 30.90 | Helped |
| Replace PDPP Res-MLP classifier with a transformer classifier (ablation) | PDPP · CrossTask T=3 | SR: 37.2 -> 39.08 | Helped |
| Use a two-layer 1D convolution encoder with ReLU instead of one layer (ablation) | MTID · CrossTask T=3 | SR: 39.71 -> 40.45 | Helped |
| Use three 1D convolution layers with ReLU instead of two (ablation) | MTID · CrossTask T=3 | SR: 40.45 -> 36.04 | Hurt |
Surprising findings
- Masking the action space during denoising iterations collapses performance (SR 3.38 vs 40.45), so masking only the initialization works.
- On COIN, the model underperforms the Deterministic baseline in uncertainty modeling, which the authors attribute to shorter action sequences.
Limitations
- Interpolated features underperform real visual features on larger datasets with longer sequences such as CrossTask, indicating difficulty with complex temporal relations.
- Memory efficiency limits model scale; larger U-Net (512) overfits on CrossTask.
- Reported mIoU computed per sequence is lower than prior batch-averaged numbers, complicating comparison.
Experimental setup
Datasets: CrossTask COIN NIV
Benchmarks: CrossTask COIN NIV
Metrics: SR mAcc mIoU KL-Div NLL ModePrec ModeRec
Baselines: Random Retrieval-Based WLTDO UAAA UPN DDN PlaTe Ext-GAIL P3IV EGPP PDPP KEPP
Builds on: PDPP DDIM KEPP SCHEMA
Closest papers by meaning
- Procedure Knowledge Decoupled Distillation Strategy for Procedure Planning in Instructional Videos (AAAI 2025 · similarity 0.89)
- Masked Diffusion Vision-Language Models for Temporal Action Localization (NeurIPS 2026 · similarity 0.87)
- SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos (ICLR 2024 · similarity 0.87)
- ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation (NeurIPS 2024 · similarity 0.86)
- Instruction-Guided Visual Masking (NeurIPS 2024 · similarity 0.86)
- ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.