Research map / Video / Video generation
EnergyMoGen: Compositional Human Motion Generation with Energy-Based Diffusion Model in Latent Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EnergyMoGen, a latent motion diffusion framework that interprets diffusion and cross-attention as energy-based models to compose multiple text concepts into human motions, fused via Synergistic Energy Fusion.
Problem
Latent diffusion models for text-driven human motion struggle to compose multiple semantic concepts (conjunction, negation, multi-concept) into one coherent motion, since latent vectors lack explicit per-frame correspondence.
Approach
Views the latent diffusion denoiser as a latent-aware energy model (classifier-free-guidance-style composition) and adds a semantic-aware cross-attention energy model with adaptive gradient descent on text embeddings; Synergistic Energy Fusion combines both with the full multi-concept text to offset text misalignment and motion distortion.
Key results
- On HumanML3D, EnergyMoGen reaches Top-1 R-Precision 0.523 and MM-Dist 2.915 vs. MLD's 0.481 and 3.196 (FID 0.188 vs. 0.473 for MLD is not best; ReMoDiffuse FID 0.103).
- On MTT, Ours + AGD + SEF reaches R-Precision Top-1 15.9 and TMR-Score 0.591 for compositional generation, vs. 12.7 Top-1 for best multi-concept baseline MotionDiffuse at 10.9.
- Fine-tuning on CompML (5000 composed motions) slightly improves HumanML3D Top-1 R-Precision from 0.523 to 0.526.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CompML motion-text pairs during finetuning (ablation) | ENERGYMOGEN · HumanML3D test set | R-Precision Top-1: 0.523±.003 -> 0.526±.003 | Helped |
| Add adaptive gradient descent of text embeddings (ablation) | ENERGYMOGEN · MTT multi-concept motion generation | R-Precision R@1: 12.7 -> 14.0 | Helped |
| Add adaptive gradient descent of text embeddings (ablation) | ENERGYMOGEN · MTT multi-concept motion generation | FID: 0.592 -> 0.587 | Helped |
| Use latent-aware composition only instead of synergistic energy fusion (ablation) | ENERGYMOGEN · MTT compositional motion generation | R-Precision R@1: 15.9 -> 9.7 | Hurt |
| Use latent-aware composition only instead of synergistic energy fusion (ablation) | ENERGYMOGEN · MTT compositional motion generation | Transition distance: 1.6 -> 1.6 | No clear effect |
| Use semantic-aware composition only instead of synergistic energy fusion (ablation) | ENERGYMOGEN · MTT compositional motion generation | R-Precision R@1: 15.9 -> 15.1 | Hurt |
| Use semantic-aware composition only instead of synergistic energy fusion (ablation) | ENERGYMOGEN · MTT compositional motion generation | Transition distance: 1.6 -> 2.2 | Hurt |
| Fuse latent-aware, semantic-aware, and single-text distributions (ablation) | ENERGYMOGEN · MTT compositional motion generation | R-Precision R@1: 9.7 / 15.1 -> 15.9 | Helped |
| Fuse latent-aware, semantic-aware, and single-text distributions (ablation) | ENERGYMOGEN · MTT compositional motion generation | Transition distance: 2.2 -> 1.6; 1.6 -> 1.6 | Mixed |
Surprising findings
- Latent-only composition performs poorly on R-Precision and TMR-Score despite lowering transition distance, while semantic-only composition causes foot sliding and jitter; only their fusion works well.
Limitations
- Semantic-only composition suffers from foot sliding and motion jitter.
- Latent-only composition suffers from text misalignment.
- Fusion weights (lambda values) are hand-set hyperparameters.
Experimental setup
Datasets: HumanML3D KIT-ML MTT CompML
Benchmarks: HumanML3D KIT-ML MTT
Metrics: R-Precision FID MM-Dist Diversity MModality TMR-Score transition distance
Baselines: MDM MotionDiffuse M2DM Fg-T2M ReMoDiffuse FineMoGen MLD GUESS MotionMamba
Builds on: Motion Latent Diffusion (MLD) classifier-free guidance energy-based cross-attention (Park et al.) modern Hopfield networks compositional visual generation via EBMs
Closest papers by meaning
- HuTuMotion: Human-Tuned Navigation of Latent Motion Diffusion Models with Minimal Feedback (AAAI 2024 · similarity 0.87)
- AMD: Autoregressive Motion Diffusion (AAAI 2024 · similarity 0.85)
- Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregression (CVPR 2025 · similarity 0.85)
- Human Motion Diffusion as a Generative Prior (ICLR 2024 · similarity 0.84)
- TIMotion: Temporal and Interactive Framework for Efficient Human-Human Motion Generation (CVPR 2025 · similarity 0.83)
- MotionMix: Weakly-Supervised Diffusion for Controllable Motion Generation (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.