Research map / Video / Video generation
Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes YODA, an unsupervised autoregressive flow-matching video generator (built on RIVER) that controls objects via sparse optical-flow inputs sampled at a few pixels, trained with randomized conditioning.
Problem
Controllable video generation of multi-object scenes from a single frame and sparse, pixel-level motion controls, learned without object annotations or segmentation.
Approach
Condition an autoregressive flow-matching video model (RIVER) on sparse optical-flow controls sampled from a flow-weighted distribution, encoded per tile with a restricted-receptive-field MLP, and randomly drop context and control during training so the model can generate from a single frame and implicitly learns object extents.
Key results
- On BAIR with 5 controls, YODA achieves LPIPS 0.112, FID 18.2, FVD 264, beating GLASS (0.118, 18.7, 411) and other baselines.
- On iPER, YODA (1st frame) reaches FVD 133.08, better than Hao et al. (235.08) and II2V (220.34) but behind iPOKE (77.50).
- Local control error on BAIR robot arm is 1.41 px for YODA vs 1.75 px for GLASS and 6.29 px for CADDY; under o.o.d. large controls YODA gets 3.37 px vs GLASS 30.06 px.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use YODA with 5 controls instead of GLASS | BAIR video generation · BAIR | LPIPS: 0.118 -> 0.112 | Helped |
| Use YODA with 5 controls instead of GLASS | BAIR video generation · BAIR | FID: 18.7 -> 18.2 | Helped |
| Use YODA with 5 controls instead of [cite] affine | BAIR video generation · BAIR | FVD: 292 -> 264 | Helped |
| Use YODA (1st frame) instead of iPOKE | iPER video generation · iPER | FVD: 77.50 -> 133.08 | Hurt |
| Use YODA (1st frame) instead of Hao w/ KP | iPER video generation · iPER | LPIPS: 0.04 -> 0.09 | Hurt |
| Use YODA (1st frame) instead of Hao w/ KP | iPER video generation · iPER | SSIM: 0.93 -> 0.86 | Hurt |
| Use YODA instead of GLASS | BAIR robot arm control · BAIR robot arm control | local error (pixels): 1.75 -> 1.41 | Helped |
| Add randomized conditioning (π=0.5) rather than no randomized conditioning (π=0) (ablation) | YODA · CLEVRER, n_c=5 | FVD: 401 -> 70 | Helped |
| Add randomized conditioning (π=0.5) rather than no randomized conditioning (π=0) (ablation) | YODA · CLEVRER, n_c=5 | LPIPS: 0.288 -> 0.126 | Helped |
| Add randomized conditioning (π=0.5) rather than no randomized conditioning (π=0) (ablation) | YODA · CLEVRER, n_c=5 | PSNR: 24.88 -> 30.07 | Helped |
| Add randomized conditioning (π=0.5) rather than no randomized conditioning (π=0) (ablation) | YODA · CLEVRER, n_c=5 | FID: 9.9 -> 5.7 | Helped |
| Use 5 control vectors instead of 1 (ablation) | YODA · CLEVRER, π=0.5 | FVD: 83 -> 70 | Helped |
Surprising findings
- Restricting the receptive field of the sparse flow encoder (tile-wise MLP) gives better independent object control than a convolutional encoder.
- Using n_c=5 controls during training outperforms n_c=100 and n_c=1 trade-off, with too many controls hurting interaction modelling.
Limitations
- Local/global error metrics are coarse and can miss rotation versus translation.
- Does not outperform iPOKE on iPER FVD (133 vs 77.5).
- Control inputs are 2D shifts at pixels, not semantic object-level actions.
Experimental setup
Datasets: CLEVRER BAIR iPER
Benchmarks: BAIR iPER CLEVRER
Metrics: LPIPS FID FVD PSNR SSIM local error global error cosine error
Baselines: MoCoGAN SAVP CADDY GLASS iPOKE II2V Hao et al.
Builds on: RIVER flow matching U-ViT VQGAN classifier-free guidance
Closest papers by meaning
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.89)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.88)
- CAGE: Unsupervised Visual Composition and Animation for Controllable Video Generation (AAAI 2025 · similarity 0.87)
- Boximator: Generating Rich and Controllable Motions for Video Synthesis (ICML 2024 · similarity 0.87)
- Move-in-2D: 2D-Conditioned Human Motion Generation (CVPR 2025 · similarity 0.87)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.