Research map / Video / Video generation
Force Prompting: Video Generation Models Can Learn And Generalize Physics-based Control Signals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds physical force control (local point pokes and global wind fields) to CogVideoX-5B-I2V via a ControlNet trained on ~15k synthetic Blender/PhysDreamer videos, showing generalization to real images without a physics simulator at inference.
Problem
Video generation models lack precise control via physical forces; paired force-video data is scarce in the real world and synthetic simulators have limited visual quality and diversity.
Approach
Encode force as a control video (a Gaussian blob moving with force magnitude/angle for point forces; a constant channel encoding magnitude and angle for wind) and train a ControlNet on a frozen image-to-video backbone using small synthetic datasets, relying on the pretrained video prior to generalize.
Key results
- Point force model wins against Motion Prompting in 2AFC human study on force adherence at 91%, 89%, 86% across linear, oscillatory, and complex motion.
- Global wind model achieves 93% force adherence win rate vs Motion Prompting on tethered motion and 90% on fluid dynamics.
- Against PhysDreamer, mean force adherence preference is 58.33% with visual quality preferred for PhysDreamer (36.67%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add distractor balls to point-force training scenes (ablation) | Point force model · Point force localization | Force localization | Helped |
| Train the wind model with varied flag quantities instead of only one flag (ablation) | Global force model · Aerodynamic and fluid motion | Generalization across materials | Helped |
| Train the wind model with diverse backgrounds instead of a single background (ablation) | Global force model · Wind benchmark | Visual quality | Helped |
| Include wind-related keywords (wind/breeze/blow) during training (ablation) | Global force model · Diverse wind scenarios | Generalization | Helped |
| Train a unified model on point-force and wind-force data (ablation) | Force Prompting · Point force benchmark | Point-force control robustness | Hurt |
| Train a unified model on point-force and wind-force data (ablation) | Force Prompting · Wind force benchmark | Background dynamics | Helped |
| Use half as many synthetic flag-waving training videos (ablation) | Global force model · Wind force generalization | Generalization | Hurt |
| Compare Force Prompting against Motion Prompting | Point force model · Linear Motion | Force adherence: 91% | Helped |
| Compare Force Prompting against Motion Prompting | Point force model · Oscillatory Motion | Force adherence: 89% | Helped |
| Compare Force Prompting against Motion Prompting | Point force model · Complex Motion | Force adherence: 86% | Helped |
| Compare Force Prompting against Motion Prompting | Global force model · Tethered Motion | Force adherence: 93% | Helped |
| Compare Force Prompting against Motion Prompting | Global force model · Aerodynamic Motion | Force adherence: 90% | Helped |
Surprising findings
- Wind model trained only on flags generalizes to aerodynamic and fluid motion (smoke, snow, steam) without training on them.
- Text keywords (wind/breeze/blow) during training substantially affect generalization, while their presence at inference matters less.
Limitations
- Force values are not calibrated to physical units; relative intuitive physics only.
- Inherits failure modes and physical understanding limits of the base video prior (e.g., hair reorientation errors).
- Visual quality sometimes trails PhysDreamer.
Experimental setup
Datasets: Blender-rendered flag videos Blender-rendered rolling ball videos PhysDreamer carnation videos Pexels benchmark images
Benchmarks: Pexels-curated local force benchmark (63 images) Pexels-curated global force benchmark (41 images) PhysDreamer benchmark plants (6 scenes)
Metrics: 2AFC human preference (force adherence) motion realism preference visual quality preference detection-based travel distance
Baselines: Text-only zero-shot CogVideoX Text-only fine-tuned Motion Prompting PhysDreamer
Builds on: CogVideoX ControlNet PhysDreamer Motion Prompting
Closest papers by meaning
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.91)
- How Far Is Video Generation from World Model: A Physical Law Perspective (ICML 2025 · similarity 0.89)
- WISA: World simulator assistant for physics-aware text-to-video generation (NeurIPS 2025 · similarity 0.87)
- Seeing the Wind from a Falling Leaf (NeurIPS 2025 · similarity 0.87)
- RoboScape: Physics-informed Embodied World Model (NeurIPS 2025 · similarity 0.86)
- PhysGen3D: Crafting a Miniature Interactive World from a Single Image (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.