Research map / Video / Video generation
OmniControl: Control Any Joint at Any Time for Human Motion Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniControl, a diffusion-based text-to-motion model that accepts sparse global spatial control signals on any joint at any time via analytic spatial guidance and a ControlNet-style realism guidance.
Problem
Text-conditioned human motion diffusion models cannot easily incorporate sparse, global spatial control signals on joints other than the pelvis because relative pose representations make inpainting-based constraints ill-posed.
Approach
Convert generated relative motion to global coordinates inside a spatial guidance loss and perturb the predicted mean iteratively via gradients, combined with a trainable ControlNet-like realism guidance copy of the transformer encoder that adds residual features to refine all joints.
Key results
- On HumanML3D pelvis control, Avg. err. drops to 0.0338 vs 0.1439 for GMD (about 79% reduction) and FID 0.218 vs 0.475 for PriorMDM.
- A single model trained on all joints achieves average control error 0.0404 on HumanML3D across pelvis, feet, head and wrists, comparable to pelvis-only control.
- Removing spatial guidance raises Avg. err. from 0.0385 to 0.4137 on HumanML3D, showing spatial guidance is critical for control accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OmniControl rather than GMD for pelvis control | HumanML3D text-conditioned motion generation · HumanML3D test, pelvis control | Avg. err.: 0.1439 -> 0.0338 | Helped |
| Use OmniControl rather than PriorMDM for pelvis control | KIT-ML text-conditioned motion generation · KIT-ML test, pelvis control | Avg. err.: 0.2305 -> 0.0759 | Helped |
| Add spatial guidance (ablation) | OmniControl · HumanML3D test, all-joint average | Avg. err.: 0.4137 -> 0.0404 | Helped |
| Add realism guidance (ablation) | OmniControl · HumanML3D test, all-joint average | FID: 0.692 -> 0.310 | Helped |
| Compute the spatial-guidance gradient with respect to the predicted mean rather than x_t (ablation) | OmniControl · HumanML3D test, all-joint average | Avg. err.: 0.2380 -> 0.0404 | Helped |
Surprising findings
- Global (non-relative) pose representation performs much worse for text-to-motion generation (FID 1.396 vs 0.544), so relative representation is kept.
- Realism guidance alone is insufficient for control accuracy, but removing it roughly doubles FID-like realism degradation (FID 0.310 to 0.692).
Limitations
- Substantial foot skating remains despite the realism guidance.
- Slow inference due to roughly 1,000 diffusion steps plus iterative guidance (about 121 s per sample).
- Conflicting control signals on multiple joints produce unnatural motions.
Experimental setup
Datasets: HumanML3D KIT-ML AMASS HumanAct12
Benchmarks: HumanML3D KIT-ML
Metrics: FID R-precision diversity foot skating ratio trajectory error location error average error
Baselines: MDM PriorMDM GMD
Builds on: MDM ControlNet classifier guidance GMD PriorMDM
Closest papers by meaning
- InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint (NeurIPS 2024 · similarity 0.86)
- Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videos (NeurIPS 2024 · similarity 0.83)
- Programmable Motion Generation for Open-Set Motion Control Tasks (CVPR 2024 · similarity 0.82)
- CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objects (NeurIPS 2025 · similarity 0.82)
- POMP: Physics-consistent Motion Generative Model through Phase Manifolds (CVPR 2025 · similarity 0.81)
- HuTuMotion: Human-Tuned Navigation of Latent Motion Diffusion Models with Minimal Feedback (AAAI 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.