Research map / Image generation / Autoregressive and latent generation
ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 578K-sequence step-by-step visual instruction dataset automatically from HowTo100M narrations and trains ShowHowTo, a video diffusion model that generates scene-consistent image sequences conditioned on an input image and step texts.
Problem
Generating step-by-step visual instructions grounded in a user's specific environment (given an input scene image and ordered textual steps) is hard, largely due to the lack of large-scale training data with aligned image-text instruction sequences.
Approach
An automatic four-stage pipeline (ASR with WhisperX, LLM-based video filtering, LLM step extraction from narrations, CLIP-based cross-modal frame alignment) mines image-text instruction sequences from HowTo100M, and a latent video diffusion model is fine-tuned to attend to the input image and a separate text prompt per step, with variable-length sequence training.
Key results
- ShowHowTo achieves scene consistency of 0.34 vs 0.13 for the best prior method (GenHowTo) on the ShowHowTo test set, with step faithfulness 0.52 and task faithfulness 0.42.
- Step faithfulness of ShowHowTo (0.52) exceeds that of the source dataset sequences (0.50) on the ShowHowTo test set.
- In a 9-participant user study, ShowHowTo was preferred over all baselines on step, scene and task criteria, and was preferred over source sequences in 42% of cases for step and scene.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a separate text prompt for each step rather than one concatenated or summarized prompt (ablation) | ShowHowTo · ShowHowTo test set | Step Faithfulness: 0.21 -> 0.52; 0.20 -> 0.52 | Helped |
| Use a separate text prompt for each step rather than one concatenated or summarized prompt (ablation) | ShowHowTo · ShowHowTo test set | Scene Consistency: 0.29 -> 0.34; 0.30 -> 0.34 | Helped |
| Use the free-form step description instead of the fixed input prompt 'an image' (ablation) | ShowHowTo · ShowHowTo test set | Step Faithfulness: 0.51 -> 0.52 | Helped |
| Train on ShowHowTo data instead of WikiHow-VGSI (ablation) | ShowHowTo · ShowHowTo test set | Scene Consistency: 0.12 -> 0.34 | Helped |
| Train on ShowHowTo data instead of HowToStep (ablation) | ShowHowTo · ShowHowTo test set | Step Faithfulness: 0.39 -> 0.52 | Helped |
| Train on ShowHowTo data instead of food-only ShowHowTo data (ablation) | ShowHowTo · ShowHowTo test set | Task Faithfulness: 0.37 -> 0.42 | Helped |
| Train on sequences up to 4 steps rather than variable-length sequences up to 8 steps (ablation) | ShowHowTo · ShowHowTo test set | Scene Consistency: 0.34 -> 0.39 | Helped |
| Train on sequences up to 4 steps rather than variable-length sequences up to 8 steps (ablation) | ShowHowTo · ShowHowTo test set | Step Faithfulness: 0.52 -> 0.47 | Hurt |
| Train on sequences up to 16 steps rather than variable-length sequences up to 8 steps (ablation) | ShowHowTo · ShowHowTo test set | Step Faithfulness: 0.52 -> 0.57 | Helped |
| Train on sequences up to 16 steps rather than variable-length sequences up to 8 steps (ablation) | ShowHowTo · ShowHowTo test set | Scene Consistency: 0.34 -> 0.26 | Hurt |
| Train on consecutive rather than randomly sampled steps (ablation) | ShowHowTo · ShowHowTo test set | Scene Consistency: 0.32 -> 0.34 | Helped |
| Train on variable-length sequences up to 8 steps rather than exactly 8 steps (ablation) | ShowHowTo · ShowHowTo test set | Scene Consistency: 0.26 -> 0.34 | Helped |
Surprising findings
- ShowHowTo's step faithfulness exceeds that of the original source video frames (0.52 vs 0.50).
- Training on shorter sequences (up to 4 frames) maximizes scene consistency but hurts step faithfulness, while longer sequences show the opposite trend.
Limitations
- Struggles to maintain object states across many frames (e.g., raw meat appearing after cooking).
- Rare objects (e.g., electrical components) may be generated in physically impossible configurations.
- Task faithfulness still lags behind source sequences, indicating room for improvement.
Experimental setup
Datasets: HowTo100M WikiHow-VGSI HowToStep WebVid10M
Benchmarks: ShowHowTo test set WikiHow-VGSI subset GenHowTo
Metrics: step faithfulness (DFN-CLIP zero-shot) scene consistency (DINOv2 retrieval) task faithfulness (DFN-CLIP zero-shot) user study win rate
Baselines: InstructPix2Pix AURORA GenHowTo Phung et al. StackedDiffusion Stable Diffusion HowToStep WikiHow-VGSI
Builds on: Stable Video Diffusion-style latent video diffusion (Blattmann et al. / Xing et HowTo100M StackedDiffusion GenHowTo WhisperX
Closest papers by meaning
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.86)
- Step Differences in Instructional Video (CVPR 2024 · similarity 0.86)
- Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos (ICLR 2025 · similarity 0.86)
- Instruction-based Image Manipulation by Watching How Things Move (CVPR 2025 · similarity 0.86)
- Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflection (ICLR 2025 · similarity 0.86)
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.