Research map / Video / Video generation
DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DART, a diffusion-based autoregressive motion-primitive model with a VAE latent space that enables real-time text-driven motion generation and spatial control via latent noise optimization or RL policies.
Problem
Generating long, continuous text-conditioned human motion online and in real time, and adding precise spatial constraints (goals, scenes) to text-conditioned motion generation.
Approach
Model motion as overlapping autoregressive motion primitives (H=2 history, F=8 future) in a compact VAE latent space, with a 10-step latent diffusion denoiser conditioned on history and CLIP text; spatial control is done by optimizing the diffusion initial noises or training PPO policies whose action is the latent noise.
Key results
- On BABEL temporal composition, DART reaches segment FID 3.79 vs 5.81 for FlowMDM (best), and transition FID 1.86, while running at 334 frames/s with 0.02 s latency versus 31 frames/s for FlowMDM.
- In human preference study, DART is preferred over FlowMDM for realism 53.3% vs 46.7% and for semantic alignment 51.3% vs 48.7%.
- Text-conditioned goal-reaching RL policy runs at 240 frames per second with 100% success rate across walk, run and hop styles.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove the variational autoencoder and diffuse in raw motion space (ablation) | DART · BABEL validation transition | PJ: 0.06$\pm$0.00 -> 0.20$\pm$0.00 | Hurt |
| remove scheduled training (ablation) | DART · BABEL validation segment | R-prec: 0.62$\pm$0.01 -> 0.39$\pm$0.01 | Hurt |
| replace motion primitives with single-frame autoregressive prediction (ablation) | DART · BABEL validation segment | FID: 3.79$\pm$0.06 -> 10.31$\pm$0.09 | Hurt |
| increase primitive future length from 8 to 16 frames (ablation) | DART · BABEL validation segment | FID: 3.79$\pm$0.06 -> 4.04$\pm$0.10 | Hurt |
| reduce diffusion steps from 10 to 2 (ablation) | DART · BABEL validation segment | FID: 3.79$\pm$0.06 -> 4.44$\pm$0.04 | Hurt |
| increase VAE KL loss weight from 1e^{-6} to 1 (ablation) | Motion primitive VAE · Test motions | Reconstruction error: 0.08 to 0.44 | Hurt |
| add auxiliary losses with weight 100 (ablation) | Motion primitive VAE · VAE reconstructions | Jerk: 3.67 to 2.45 | Helped |
| use DART instead of FlowMDM | Text-conditioned temporal motion composition · BABEL validation segment | FID: 5.81$\pm$0.10 -> 3.79$\pm$0.06 | Helped |
| use DART instead of FlowMDM | Text-conditioned temporal motion composition · BABEL validation segment | R-prec: 0.67$\pm$0.00 -> 0.62$\pm$0.01 | Hurt |
| use DART instead of FlowMDM | Text-conditioned temporal motion composition · BABEL validation (5000-frame sequence) | Speed (frame/s): 31$\pm$0 -> 334 $\pm$2 | Helped |
| use DART instead of DNO | Text-conditioned motion in-between · Text-conditioned motion in-between | Goal error (cm): 4.24 $\pm$ 1.34 -> 0.59 $\pm$ 0.01 | Helped |
| use DART walk policy instead of GAMMA walk | Text-conditioned goal-reaching controller · Goal-reaching (walk) | Success rate: $0.95 \pm 0.03$ -> $\mathbf{1.0} \pm 0.0$ | Helped |
Surprising findings
- Only 10 diffusion steps suffice; reducing to fewer than 10 barely hurts performance, while 2 steps raises FID notably.
- Frame-by-frame prediction (H=1, F=1) performs much worse than the H=2, F=8 primitive for text-conditioned learning.
Limitations
- Trained on coarse sentence-level labels (HumanML3D), text-motion alignment degrades for multi-action texts.
- Online generation introduces transition delay that lowers R-precision relative to offline FlowMDM.
Experimental setup
Datasets: BABEL HumanML3D AMASS
Benchmarks: BABEL validation set HumanML3D test sequences
Metrics: FID R-precision MM-Dist diversity peak jerk area under jerk speed (frame/s) latency memory history error
Baselines: TEACH DoubleTake T2M-GPT* FlowMDM DNO OmniControl GAMMA
Builds on: MLD DDPM classifier-free guidance PPO FlowMDM
Closest papers by meaning
- OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers (CVPR 2024 · similarity 0.89)
- AMD: Autoregressive Motion Diffusion (AAAI 2024 · similarity 0.89)
- Enhanced Fine-Grained Motion Diffusion for Text-Driven Human Motion Synthesis (AAAI 2024 · similarity 0.88)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.88)
- SnapMoGen: Human Motion Generation from Expressive Texts (NeurIPS 2025 · similarity 0.88)
- Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregression (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.