AI Paper Atlas

Research map / Video / Video generation

Efficient Pre-training for Localized Instruction Generation of Procedural Videos

Anil Batra, Davide Moltisanti, Laura Sevilla-Lara, Marcus Rohrbach, Frank Keller

ECCV 2024 · Video › Video generation · approach: Compositional text-to-video diffusion · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
pre-train on Sieve&Swap captions (0.51M segments) instead of raw ASR (2.75M) (ablation)ProcX · YouCook2 (finetuned)SODA-C: 6.97 -> 7.29Improved
pre-train on Sieve&Swap captions instead of raw ASR (ablation)Vid2Seq · YouCook2 (finetuned)SODA-C: 3.65 -> 4.21Improved
pre-train on Sieve&Swap captions instead of raw ASR (ablation)PDVC · YouCook2 (finetuned)SODA-C: 5.81 -> 6.16Improved
pre-train on Sieve&Swap instead of raw ASR, zero-shot (ablation)ProcX · YouCook2 zero-shotSODA-C: 1.10 -> 2.03Improved
pre-train on Sieve&Swap instead of raw ASR, zero-shot localization on Tasty (ablation)ProcX · Tasty zero-shotSODA-D: 35.34 -> 24.97Hurt
replace LSTM captioner with contrastive Transformer captioner, no gVFL (ablation)PDVC · YouCook2SODA-C: 5.43 -> 4.94Hurt
add gIoU-guided varifocal loss (gVFL) to Transformer captioner (ablation)PDVC with Transformer captioner · YouCook2SODA-C: 4.94 -> 5.84Improved
add key-aware deformable attention to encoder and decoder (ablation)ProcX · YouCook2SODA-D: 33.33 -> 34.63Improved
add key-aware deformable attention to caption metrics (ablation)ProcX · YouCook2METEOR: 5.34 -> 5.35No clear effect
use ProcX architecture instead of PDVC without pre-trainingPDVC · Tasty (test)SODA-C: 7.18 -> 10.83Improved
use CLIP features instead of S3D features for PDVC with Sieve&Swap pre-trainingPDVC · YouCook2 (val)SODA-C: 6.16 -> 5.48Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy