Research map / Video / Video generation
Efficient Pre-training for Localized Instruction Generation of Procedural Videos
Anil Batra, Davide Moltisanti, Laura Sevilla-Lara, Marcus Rohrbach, Frank Keller
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train on Sieve&Swap captions (0.51M segments) instead of raw ASR (2.75M) (ablation) | ProcX · YouCook2 (finetuned) | SODA-C: 6.97 -> 7.29 | Improved |
| pre-train on Sieve&Swap captions instead of raw ASR (ablation) | Vid2Seq · YouCook2 (finetuned) | SODA-C: 3.65 -> 4.21 | Improved |
| pre-train on Sieve&Swap captions instead of raw ASR (ablation) | PDVC · YouCook2 (finetuned) | SODA-C: 5.81 -> 6.16 | Improved |
| pre-train on Sieve&Swap instead of raw ASR, zero-shot (ablation) | ProcX · YouCook2 zero-shot | SODA-C: 1.10 -> 2.03 | Improved |
| pre-train on Sieve&Swap instead of raw ASR, zero-shot localization on Tasty (ablation) | ProcX · Tasty zero-shot | SODA-D: 35.34 -> 24.97 | Hurt |
| replace LSTM captioner with contrastive Transformer captioner, no gVFL (ablation) | PDVC · YouCook2 | SODA-C: 5.43 -> 4.94 | Hurt |
| add gIoU-guided varifocal loss (gVFL) to Transformer captioner (ablation) | PDVC with Transformer captioner · YouCook2 | SODA-C: 4.94 -> 5.84 | Improved |
| add key-aware deformable attention to encoder and decoder (ablation) | ProcX · YouCook2 | SODA-D: 33.33 -> 34.63 | Improved |
| add key-aware deformable attention to caption metrics (ablation) | ProcX · YouCook2 | METEOR: 5.34 -> 5.35 | No clear effect |
| use ProcX architecture instead of PDVC without pre-training | PDVC · Tasty (test) | SODA-C: 7.18 -> 10.83 | Improved |
| use CLIP features instead of S3D features for PDVC with Sieve&Swap pre-training | PDVC · YouCook2 (val) | SODA-C: 6.16 -> 5.48 | Hurt |
Closest papers by meaning
- GePSAn: Generative Procedure Step Anticipation in Cooking Videos (ICCV 2023 · similarity 0.88)
- VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning (ICLR 2025 · similarity 0.86)
- ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions (CVPR 2025 · similarity 0.85)
- SLVMBench: Skill Learning from Video Memory (NeurIPS 2026 · similarity 0.85)
- Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos (ICLR 2025 · similarity 0.85)
- Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
