Research map / Image generation / Text-to-image diffusion
OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OMG, a pretrain-then-finetune text-to-motion model that pretrains a diffusion transformer up to 1B parameters on 20M+ unlabeled motion frames and adds a motion ControlNet with Mixture-of-Controllers for open-vocabulary text conditioning.
Problem
Text-to-motion generation fails or produces implausible motion for unseen open-vocabulary text because paired text-motion data is scarce and text-to-motion mapping is many-to-many.
Approach
Scale an unconditional motion diffusion transformer on large unlabeled motion data, then add a frozen-backbone ControlNet whose Mixture-of-Controllers block uses cross-attention and text-token-specific experts to inject per-token motion residuals aligned with CLIP token embeddings.
Key results
- On the zero-shot Mixamo set, OMG reaches FID 1.164 versus 1.229 for MLD and 1.297 for MDM, and CLIP-score 0.588 versus 0.556 for MLD.
- On in-domain HumanML3D, OMG gets the best R-Precision (0.784) but its FID (0.381) is worse than MLD (0.473) and T2M-GPT (0.116)... among diffusion methods it has the best FID.
- Ablations show that removing the zero convolution or the attention mask, or replacing the MoC block with cross-attention plus FFN, each worsens FID; FID drops as expert pool size grows from 1 to 12.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add large-scale motion pre-training (ablation) | OMG · Mixamo test set (zero-shot) | FID | Helped |
| increase model size (ablation) | OMG with pre-training · Mixamo test set (zero-shot) | FID | Helped |
| increase the expert pool size (ablation) | OMG · Mixamo test set (zero-shot) | FID | Helped |
| retain the full MoC block instead of using only cross-attention and an FFN (ablation) | OMG · Mixamo test set (zero-shot) | FID: 1.252±.006 -> 1.164±.009 | Helped |
| retain the full MoC block instead of using only cross-attention and an FFN (ablation) | OMG · Mixamo test set (zero-shot) | CLIP-score: 0.552±.003 -> 0.588±.002 | Helped |
| use zero convolutions instead of Gaussian-initialized standard convolutions (ablation) | OMG · Mixamo test set (zero-shot) | FID: 1.339±.005 -> 1.164±.009 | Helped |
| use zero convolutions instead of Gaussian-initialized standard convolutions (ablation) | OMG · Mixamo test set (zero-shot) | CLIP-score: 0.535±.004 -> 0.588±.002 | Helped |
| retain the attention mask on expert outputs (ablation) | OMG · Mixamo test set (zero-shot) | FID: 1.246±.008 -> 1.164±.009 | Helped |
| retain the attention mask on expert outputs (ablation) | OMG · Mixamo test set (zero-shot) | CLIP-score: 0.557±.003 -> 0.588±.002 | Helped |
Surprising findings
- On in-domain HumanML3D, OMG's FID (0.381) is worse than T2M-GPT's (0.116), so the advantage is mainly zero-shot.
Limitations
- Zero-shot gains are measured only on Mixamo, a single out-of-domain set.
- In-domain FID on HumanML3D is weaker than some baselines such as T2M-GPT.
Experimental setup
Datasets: HumanML3D Mixamo over 13 public motion capture and character animation datasets (20M+ frames)
Benchmarks: HumanML3D Mixamo (zero-shot)
Metrics: FID R-Precision CLIP-score Diversity
Baselines: MotionCLIP MAA MotionDiffuse MDM MLD T2M-GPT MotionGPT
Builds on: Diffusion Transformer (DiT) ControlNet Motion Diffusion Model (MDM) CLIP Mixture-of-Experts
Closest papers by meaning
- SnapMoGen: Human Motion Generation from Expressive Texts (NeurIPS 2025 · similarity 0.90)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.89)
- MMM: Generative Masked Motion Model (CVPR 2024 · similarity 0.89)
- DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Control (ICLR 2025 · similarity 0.89)
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.88)
- Enhanced Fine-Grained Motion Diffusion for Text-Driven Human Motion Synthesis (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.