Research map / Video / Video generation
Programmable Motion Generation for Open-Set Motion Control Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes programmable motion generation, which compiles arbitrary motion control tasks into differentiable error functions from atomic constraints and optimizes the latent noise of a frozen pre-trained MDM to satisfy them.
Problem
Character animation needs arbitrary, customizable combinations of constraints (trajectories, keyframes, interactions, physics), but existing methods handle only a fixed set of tasks with task-specific data and networks (close-set motion control).
Approach
Break any control task into atomic constraints, program them as a differentiable error function (with logical operators redefined as error arithmetic), and optimize the latent noise of a frozen pre-trained motion diffusion model to minimize it, with LLMs able to write the error code automatically.
Key results
- On HSI-1 head-height constraint, ours reaches C.Err 0.012 and unsuccess rate 0.088 with FID 0.556, while IK gets high Max Acc (0.414) and MDM Edit has unsuccess rate 0.645.
- On unseen task HOI-1 (moving object), ours achieves Foot Skate 0.114 and Max Acc 0.068 versus IK's Max Acc 0.919 and PriorMDM's Foot Skate 0.327.
- Automatic programming with GPT translates task descriptions such as touching a wall, avoiding overhead barriers and walking inside a square into correct error-function code (qualitative).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize MDM latent noise against programmed constraints rather than directly optimizing joint positions with IK | Motion generation with MDM prior · HSI-1: head height constraint | Max Acc.: 0.414 -> 0.094 | Helped |
| Optimize MDM latent noise against programmed constraints rather than directly optimizing joint positions with IK | Motion generation with MDM prior · HSI-3: walking inside a square | Max Acc.: 0.292 -> 0.093 | Helped |
| Optimize MDM latent noise against programmed constraints rather than directly optimizing joint positions with IK | Motion generation with MDM prior · HOI-1: moving object | Foot Skate: 0.408 -> 0.114 | Helped |
| Use the MDM motion prior instead of IK+Reg. for the head-height task | Head height constraint generation · HSI-1: head height constraint | Foot Skate: 0.269 -> 0.075 | Helped |
| Use the MDM motion prior instead of IK+Reg. for the head-height task | Head height constraint generation · HSI-1: head height constraint | FID: 0.782 -> 0.556 | Helped |
| Add an L2-norm regularization term to IK for frame consistency | IK · HSI-1: head height constraint | Foot Skate: 0.093 -> 0.269 | Hurt |
| Recover motions from local joint positions rather than the MDM prior output | MDM Edit (Position) · Head height constraint | Bone Length Incorrect Ratio: 0.048 -> 0.525 | Hurt |
| Use latent-noise optimization with the MDM prior rather than IK for the wall-contact task | GEO-1 hand touching wall · GEO-1: hand touching wall | Max Acc.: 0.187 -> 0.104 | Helped |
| Use latent-noise optimization with the MDM prior rather than IK for the barrier task | HSI-2 avoiding barrier · HSI-2: avoiding barrier | Max Acc.: 1.919 -> 0.150 | Helped |
Surprising findings
- Inpainting-based methods (MDM Edit, PriorMDM) cannot handle inequality constraints or whole-body constraints such as center of mass.
- The method shows emergent new behaviors for unseen tasks, e.g., arms brought together and shoulders shrunk when walking between two walls.
Limitations
- Evaluation uses a fixed set of pre-defined sub-tasks and a single base model (MDM), so generality across generators is unshown.
- Semantic metrics (FID, R-precision) only apply to constraints drawn from ground-truth distribution; unseen constraints rely on non-semantic metrics.
- Current framework targets body-level motion, not whole-body detail; automatic constraint generation in rich scenes is left open.
Experimental setup
Datasets: HumanML3D
Benchmarks: HumanML3D
Metrics: Foot Skate Max Acc. C.Err Unsucc. Rate FID Diversity R-precision
Baselines: Inverse Kinematics (IK) IK+Reg. MDM Edit PriorMDM
Builds on: MDM PriorMDM DDIM Adam
Closest papers by meaning
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.85)
- InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint (NeurIPS 2024 · similarity 0.85)
- Sitcom-Crafter: A Plot-Driven Human Motion Generation System in 3D Scenes (ICLR 2025 · similarity 0.85)
- OptiWorld: Optimal Control for Video World Generation under Physical Constraints (NeurIPS 2026 · similarity 0.85)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.85)
- MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.