Research map / Video / Video generation
HuTuMotion: Human-Tuned Navigation of Latent Motion Diffusion Models with Minimal Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts the latent prior of MLD for text-to-motion by optimizing per-representative-text latents with few-shot human rankings via a zeroth-order rank-based optimizer, then sampling from a Gaussian around the nearest representative text's optimal latent.
Problem
Text-to-motion generation from latent diffusion models samples from a standard normal prior, which fails to capture data characteristics and nuanced semantics (e.g., style, personalization), and existing RLHF-style methods need large human-ranked datasets.
Approach
Replace the standard normal latent prior with per-representative-text optimal latent distributions found via zeroth-order, rank-based optimization driven by minimal online human feedback, and route each input text to its nearest representative by CLIP cosine similarity.
Key results
- On HumanML3D, Ours achieves FID 0.295 vs MLD 0.473 and R-Precision Top-3 0.785, with the best Diversity 9.828 and MM Dist 3.093 vs MLD 3.196.
- On KIT, Ours achieves FID 0.224 vs MLD 0.404 and best R-Precision Top-3 0.768 and Diversity 10.96.
- Ours* (K-means centroid texts) reaches FID 0.201 on KIT and 0.224 on HumanML3D (Ours*), showing few-shot feedback performs comparably to extensive feedback per the authors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adjust the latent prior using human feedback and semantically guided sampling | HuTuMotion with MLD · HumanML3D test | FID: 0.473 -> 0.295 | Helped |
| adjust the latent prior using human feedback and semantically guided sampling | HuTuMotion with MLD · KIT test | FID: {0.404}^{\pm.027} -> 0.224^{\pm.045} | Helped |
| adjust the latent prior using human feedback and semantically guided sampling | HuTuMotion with MLD · HumanML3D test | R Precision Top 1: 0.481 -> 0.497 | Helped |
| adjust the latent prior using human feedback and semantically guided sampling | HuTuMotion with MLD · KIT test | R Precision Top 1: {0.390}^{\pm.008} -> 0.411^{\pm.005} | Helped |
| adjust the latent prior using human feedback and semantically guided sampling | HuTuMotion with MLD · HumanML3D test | Diversity: 9.724 -> 9.828 | Helped |
| adjust the latent prior using human feedback and semantically guided sampling | HuTuMotion with MLD · KIT test | Diversity: $10.80$^{\pm.117} -> 10.96^{\pm0.090} | Helped |
| vary σ from 0.1 to 0.5 (ablation) | HuTuMotion · HumanML3D test | FID: 0.258^{\pm.005}, 0.295^{\pm.006}, 0.356^{\pm.007}, 0.437^{\pm.006}, 0.554^{\pm.010} | Mixed |
| increase the number of representative texts from 1 to 50 (ablation) | HuTuMotion · HumanML3D | R Precision Top 1 | No clear effect |
| use few-shot rather than extensive human feedback | HuTuMotion · Human motion generation | performance | No clear effect |
Surprising findings
- Using ChatGPT-generated representative texts yields comparable results to K-means centroid texts from training data.
- Increasing the number of representative texts does not guarantee performance improvement.
Limitations
- Limited to latent motion diffusion (MLD); not applicable to explicit models like MDM due to high dimensionality.
- MLD tends to miss actions for long prompts with many action descriptions, and the method only partially mitigates this.
- MModality is sensitive to sigma and higher sigma yields some semantically incorrect motions.
Experimental setup
Datasets: HumanML3D KIT
Benchmarks: HumanML3D KIT
Metrics: R Precision FID MM Dist Diversity MModality
Baselines: Seq2Seq LJ2P T2G Hier TEMOS T2M MDM MLD
Builds on: MLD DDIM zeroth-order optimization
Closest papers by meaning
- TIMotion: Temporal and Interactive Framework for Efficient Human-Human Motion Generation (CVPR 2025 · similarity 0.88)
- Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregression (CVPR 2025 · similarity 0.88)
- Human Motion Diffusion as a Generative Prior (ICLR 2024 · similarity 0.88)
- EnergyMoGen: Compositional Human Motion Generation with Energy-Based Diffusion Model in Latent Space (CVPR 2025 · similarity 0.87)
- AMD: Autoregressive Motion Diffusion (AAAI 2024 · similarity 0.87)
- Seamless Human Motion Composition with Blended Positional Encodings (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.