Research map / Video / Video generation
Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Diagnoses why VQ-based motion generators beat diffusion models, then proposes MARDM: a diffusion model with essential-only motion latents, masked autoregression with per-latent diffusion MLPs, and new essential-dimension evaluators.
Problem
Diffusion-based text-to-motion generation lags behind VQ-based methods on standard metrics, and the reasons (data representation with redundant dimensions and an evaluator that weights those dimensions) are unclear.
Approach
Remove redundant motion feature groups, encode motion with a deterministic 1D ResNet autoencoder, and train a masked autoregressive diffusion model whose per-latent diffusion MLPs are conditioned by a transformer, with velocity or noise prediction; evaluate with evaluators trained only on essential dimensions.
Key results
- On HumanML3D, Ours-SiT reaches FID 0.114 and Top-3 R-Precision 0.795, versus 0.116 and 0.786 for MoMask (the strongest VQ baseline reported).
- On KIT-ML, Ours-SiT reaches FID 0.242 versus 0.411 for MoMask and Top-1 R-Precision 0.387 versus 0.369.
- Ablation on HumanML3D: removing motion representation reformation raises FID from 0.116 to 2.196; removing autoregression raises FID to 0.551.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add motion representation reformation (ablation) | Ours · HumanML3D | FID: 2.196 -> 0.116 | Helped |
| add motion representation reformation (ablation) | Ours · HumanML3D | R-Precision Top 3: 0.703 -> 0.790 | Helped |
| add autoregressive modeling (ablation) | Ours · HumanML3D | FID: 0.551 -> 0.116 | Helped |
| add autoregressive modeling (ablation) | Ours · HumanML3D | R-Precision Top 3: 0.732 -> 0.790 | Helped |
| train with redundant motion dimensions (ablation) | T2M-GPT · HumanML3D | FID: 0.418 -> 0.335 | Helped |
| train with redundant motion dimensions (ablation) | MoMask · HumanML3D | FID: 0.200 -> 0.116 | Helped |
| predict noise instead of x0 with a cosine beta schedule (ablation) | MDM-50Step · HumanML3D | FID: 0.518 -> 31.265 | Hurt |
| evaluate using essential dimensions instead of all dimensions (ablation) | HumanML3D evaluator · HumanML3D | FID: 0.481 -> 0.518 | Mixed |
Surprising findings
- Under the original all-dimension evaluator, diffusion baselines such as MDM score much worse than VQ methods, but the gap largely closes when evaluated on essential dimensions only.
- Training VQ models with redundant dimensions improves their reconstruction and generation, the opposite of what one might expect from removing redundancy.
Limitations
- Evaluated only on HumanML3D and KIT-ML motion-language benchmarks.
- Relies on new evaluators trained on essential dimensions, so scores are not directly comparable to prior work on the original evaluator.
Experimental setup
Datasets: HumanML3D KIT-ML
Benchmarks: HumanML3D KIT-ML
Metrics: FID R-Precision Matching score MultiModality CLIP-score
Baselines: T2M-GPT MMM MoMask MDM MotionDiffuse MLD ReMoDiffuse
Builds on: MAR SiT MoMask T2M-GPT MDM
Closest papers by meaning
- AMD: Autoregressive Motion Diffusion (AAAI 2024 · similarity 0.91)
- Human Motion Diffusion as a Generative Prior (ICLR 2024 · similarity 0.90)
- Seamless Human Motion Composition with Blended Positional Encodings (CVPR 2024 · similarity 0.90)
- Auto-Regressive Diffusion for Generating 3D Human-Object Interactions (AAAI 2025 · similarity 0.89)
- MMM: Generative Masked Motion Model (CVPR 2024 · similarity 0.89)
- MotionMix: Weakly-Supervised Diffusion for Controllable Motion Generation (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.