Research map / Video / Action recognition
GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mines ~200k frame triplets from instructional videos and fine-tunes a ControlNet-style Stable Diffusion model (GenHowTo) to generate action and final-state images from an input image plus text prompt.
Problem
Generating temporally consistent, physically plausible images of actions and object state transformations that preserve the scene of an input image, conditioned on text prompts.
Approach
Automatically mine consecutive-frame triplets (initial, action, final state) from instructional videos using a self-supervised localization model, caption them with BLIP-2, and train a ControlNet-based diffusion model with semantic image conditioning (VAE-encoded input image fed into the ControlNet encoder) and image-initialized DDIM sampling, using separate models for actions and final states.
Key results
- GenHowTo reaches 0.66 action and 0.74 final-state classification accuracy on categories unseen in training, vs 0.60/0.61 for Edit Friendly DDPM and 0.55/0.63 for InstructPix2Pix.
- On seen categories, GenHowTo reaches 0.77 (actions) and 0.88 (final states), against 0.96/0.97 for real images.
- GenHowTo is the only method with FID for actions (181) below the initial-state baseline (192).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train one joint model for actions and final states instead of separate models (ablation) | GenHowTo · ChangeIt held-out categories | final-state classification accuracy: 0.88 -> 0.79 | Hurt |
| Train one joint model for actions and final states instead of separate models (ablation) | GenHowTo · ChangeIt held-out categories | action classification accuracy: 0.77 -> 0.76 | No clear effect |
| Train only on action examples (ablation) | GenHowTo · ChangeIt held-out categories | action classification accuracy | No clear effect |
| Train only on final-state examples (ablation) | GenHowTo · ChangeIt held-out categories | final-state classification accuracy | No clear effect |
| Use input-image latent initialization rather than Gaussian noise for DDIM sampling (ablation) | GenHowTo · Image generation; scene preservation | scene consistency | Helped |
| Train on instructional-video object transformations rather than relying on the compared general-purpose editing methods | GenHowTo · ChangeIt test set | action FID: 210 -> 181 | Helped |
| Train on instructional-video object transformations rather than relying on the compared general-purpose editing methods | GenHowTo · ChangeIt test set | final-state FID: 169 -> 158 | Helped |
Surprising findings
- A single joint model for actions and final states performs nearly as well on actions but much worse on final states, attributed to reduced visual fidelity.
Limitations
- Inherits biases and limitations of Stable Diffusion; can produce artistic style changes.
- Weak spatiotemporal alignment in mined triplets leads to failures on fast-moving objects and faces.
- Struggles with large scene changes in long-term generation, with compounding artifacts.
Experimental setup
Datasets: COIN ChangeIt
Benchmarks: ChangeIt held-out categories test set (233 manually verified tuples)
Metrics: classification accuracy via linear classifier on generated i FID
Baselines: Stable Diffusion Edit Friendly DDPM InstructPix2Pix CLIP zero-shot
Builds on: Stable Diffusion ControlNet Souček et al. self-supervised temporal localization BLIP-2 DDIM
Closest papers by meaning
- SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos (ICLR 2024 · similarity 0.83)
- Watch and Learn: Learning to Use Computers from Online Videos (CVPR 2026 · similarity 0.82)
- ExpertAF: Expert Actionable Feedback from Video (CVPR 2025 · similarity 0.82)
- Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos (ICLR 2025 · similarity 0.82)
- ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions (CVPR 2025 · similarity 0.81)
- Procedure Knowledge Decoupled Distillation Strategy for Procedure Planning in Instructional Videos (AAAI 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.