Research map / Image generation / Editing and style transfer
Arbitrary Motion Style Transfer with Multi-condition Motion Latent Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MCM-LDM, a motion latent diffusion model that conditions denoising on content (primary), trajectory and style (secondary) features for arbitrary motion style transfer, trained by self-reconstruction on HumanML3D.
Problem
Arbitrary motion style transfer must preserve content and trajectory while injecting a new style; prior copy-based methods transfer the content trajectory directly, causing foot sliding and weak style.
Approach
Treats style transfer as self-reconstruction in a motion VAE latent space, with a Multi-condition Denoiser that concatenates content as a primary condition and injects trajectory and style as secondary conditions via AdaLN-Zero, so trajectory is learned rather than copied.
Key results
- Achieves FMD 27.69 vs 42.68 for 1DConv+AdaIN and 113.31 for Motion Puzzle, the lowest FMD in Table 1.
- Best SRA at 58.00% and lowest FSF among non-reconstructing methods (1.28 vs 2.05 for 1DConv+AdaIN), with TSI 0.40.
- User study mean scores of 4.48 realism, 4.45 content preservation, and 4.43 style performance, all significantly above baselines (p<0.01).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MCM-LDM instead of the compared methods | motion style transfer · HumanML3D test set | FMD: 42.68 -> 27.69; 129.44 -> 27.69; 113.31 -> 27.69 | Helped |
| use MCM-LDM instead of the compared methods | motion style transfer · HumanML3D test set | SRA (%): 57.00 -> 58.00; 17.66 -> 58.00; 46.33 -> 58.00 | Helped |
| remove StyleRemover from the content encoder (ablation) | MCM-LDM · HumanML3D test set | SRA (%): 16.88 -> 58.00 | Hurt |
| remove StyleRemover from the content encoder (ablation) | MCM-LDM · HumanML3D test set | CRA (%): 93.43 -> 35.75 | Helped |
| remove trajectory condition ft (ablation) | MCM-LDM · HumanML3D test set | TSI: 0.93 -> 0.40 | Hurt |
| remove trajectory condition ft (ablation) | MCM-LDM · HumanML3D test set | SRA (%): 58.00 -> 65.11 | Helped |
| replace pre-trained MotionCLIP with a transformer encoder (ablation) | MCM-LDM style extraction · HumanML3D test set | FMD: 27.69 -> 138.55 | Hurt |
| use concatenation instead of AdaLN-Zero to fuse secondary conditions (ablation) | MCM-LDM denoiser · HumanML3D test set | FMD: 27.69 -> 32.56 | Hurt |
| use AdaIN instead of AdaLN-Zero to fuse secondary conditions (ablation) | MCM-LDM denoiser · HumanML3D test set | FMD: 27.69 -> 33.43 | Hurt |
| make style fs the primary condition instead of content fc (ablation) | MCM-LDM denoiser · HumanML3D test set | SRA (%): 58.00 -> 58.44 | No clear effect |
| make trajectory ft the primary condition instead of content fc (ablation) | MCM-LDM denoiser · HumanML3D test set | FMD: 27.69 -> 32.81 | Hurt |
Surprising findings
- Removing the StyleRemover raises CRA to 93.43 but drops SRA to 16.88, so the model collapses into reconstructing content.
- STGCN+AdaIN has the best CRA and TSI but the lowest SRA (17.66), since it mostly reconstructs the content motion.
Limitations
- Cannot generate animations with arbitrary trajectories different from the content motion.
- Less effective on motions extending beyond the training dataset's temporal scope.
- Struggles with content actions involving intricate environment interactions.
Experimental setup
Datasets: HumanML3D
Benchmarks: HumanML3D test subsets
Metrics: FMD CRA SRA TSI FSF user study realism/content/style
Baselines: 1DConv+AdaIN STGCN+AdaIN Motion Puzzle
Builds on: MLD (motion latent diffusion) MotionCLIP ArtFusion classifier-free guidance AdaLN-Zero
Closest papers by meaning
- Scalable Motion Style Transfer with Constrained Diffusion Generation (AAAI 2024 · similarity 0.88)
- Enhanced Fine-Grained Motion Diffusion for Text-Driven Human Motion Synthesis (AAAI 2024 · similarity 0.87)
- Towards Detailed Text-to-Motion Synthesis via Basic-to-Advanced Hierarchical Diffusion Model (AAAI 2024 · similarity 0.87)
- Single Motion Diffusion (ICLR 2024 · similarity 0.87)
- AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (ICLR 2024 · similarity 0.87)
- Human Motion Diffusion as a Generative Prior (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.