Research map / Video / Video generation
Optical-Flow Guided Prompt Optimization for Coherent Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MotionPrompt, which trains an optical-flow discriminator and uses it to guide on-the-fly learnable-token prompt optimization during sampling of text-to-video diffusion models to improve temporal consistency.
Problem
Text-to-video diffusion models often lack temporal consistency, and existing guidance methods (e.g., DPS) are costly or unstable when applied across all frames of a video.
Approach
Instead of guiding the latent directly, optimize learnable prompt token embeddings at inference using gradients from an optical-flow discriminator applied to a pair of decoded frames, exploiting the fact that the text prompt influences all frames at once.
Key results
- On AnimateDiff, MotionPrompt raises subject consistency from 0.9488 to 0.9528 and motion smoothness from 0.9578 to 0.9599 on VBench.
- On Lavie, motion smoothness rises from 0.9690 to 0.9765 and temporal flickering from 0.9487 to 0.9625, at the cost of dynamic degree (0.5150 to 0.3963).
- In an A/B user study with 100 participants, the method was preferred over AnimateDiff 66.5% win vs 15.7% lose, and over Lavie 55.1% win vs 23.8% lose.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MotionPrompt | Lavie · VBench | Motion smoothness: 0.9690 -> 0.9765 | Helped |
| Add MotionPrompt | AnimateDiff · VBench | Motion smoothness: 0.9578 -> 0.9599 | Helped |
| Add MotionPrompt | VideoCrafter2 · VBench | Motion smoothness: 0.9750 -> 0.9759 | Helped |
| Add MotionPrompt | Lavie · VBench | Dynamic degree: 0.5150 -> 0.3963 | Hurt |
| Add MotionPrompt | AnimateDiff · VBench | Subject consistency: 0.9488 -> 0.9528 | Helped |
| Add MotionPrompt | VideoCrafter2 · VBench | Background consistency: 0.9559 -> 0.9774 | Helped |
| Add discriminator loss (λ1) (ablation) | MotionPrompt on AnimateDiff, with λ2 / λ3 = 5.0 / 10.0 · VBench | Motion smoothness: 0.9643 -> 0.9599 | No clear effect |
| Add TV loss (λ2) (ablation) | MotionPrompt on AnimateDiff, with λ1 / λ3 = 1.0 / 10.0 · VBench | Motion smoothness: 0.9648 -> 0.9599 | Hurt |
| Increase λ2 from 5.0 to 10.0 (ablation) | MotionPrompt on AnimateDiff · VBench | Motion smoothness: 0.9599 -> 0.9658 | Helped |
| Remove token-embedding regularization (λ3 = 0.0) (ablation) | MotionPrompt on AnimateDiff · VBench | Overall consistency: 0.2529 -> 0.2517 | Hurt |
| Use 1 optimization iteration instead of 3 (ablation) | MotionPrompt on AnimateDiff · VBench | Motion smoothness: 0.9599 -> 0.9593 | Hurt |
| Use 7 optimization iterations instead of 3 (ablation) | MotionPrompt on AnimateDiff · VBench | Dynamic degree: 0.4125 -> 0.3950 | Hurt |
Surprising findings
- A discriminator trained on videos from a different source model (Lavie or VideoCrafter2) generally improved AnimateDiff results more than the matched discriminator did.
- Videos with higher subject consistency showed less token embedding change during optimization, so the method intervenes less on already-realistic videos.
Limitations
- Requires slightly more generation time than the base models.
- The optical-flow objective is not grounded in physics, so improvements may not yield physically plausible motion.
- Reduces dynamic degree on some models (e.g., Lavie drops from 0.5150 to 0.3963), showing a consistency-dynamics trade-off.
Experimental setup
Datasets: VBench prompts DAVIS WebVid
Benchmarks: VBench
Metrics: subject consistency background consistency temporal flickering motion smoothness dynamic degree overall consistency A/B user preference
Baselines: Lavie AnimateDiff VideoCrafter2 FreeInit UniCtrl VideoGuide
Builds on: diffusion posterior sampling classifier-free guidance DDIM minority-sampling prompt optimization (Um and Ye)
Closest papers by meaning
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.90)
- FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation (NeurIPS 2025 · similarity 0.89)
- LAMP: Learn A Motion Pattern for Few-Shot Video Generation (CVPR 2024 · similarity 0.89)
- Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation (ICLR 2025 · similarity 0.89)
- Make Pixels Dance: High-Dynamic Video Generation (CVPR 2024 · similarity 0.88)
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.