Research map / Image generation / Editing and style transfer
Scalable Motion Style Transfer with Constrained Diffusion Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KMCGs, a keyframe-based manifold constraint gradient applied during DDIB inference, enabling scalable motion style transfer with independently trained diffusion models per style.
Problem
Motion style transfer systems based on CycleGAN/StarGAN require joint training across style domains, which scales poorly to many styles and conflicts with data privacy; existing DDIB-style independent training loses content on complex dance motions.
Approach
Train each style's diffusion model independently and transfer via DDIB latent bridging, while imposing keyframe manifold constraint gradients from the source motion during backward diffusion to preserve content, avoiding extra training.
Key results
- KMCGs (DDIBs-gradient) reduces FPD to 0.1208 from 0.2904 for DDIBs-vanilla and 0.4138 for StarDance, indicating better content preservation.
- FMD remains comparable across methods (0.1214 for DDIBs-gradient vs 0.1295 for DDIBs-vanilla), so transfer strength is not compromised.
- Human study (21 participants) finds no significant difference in naturalness and transfer strength; body shape preservation significantly improved (p=0.0000961).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add keyframe manifold constraint gradients during transfer (ablation) | DDIBs-vanilla · AIST++ dance | FPD: 0.2904 -> 0.1208 | Helped |
| add keyframe manifold constraint gradients during transfer (ablation) | DDIBs-vanilla · AIST++ dance | FMD: 0.1295 -> 0.1214 | Helped |
| incorporate source keyframes through cross-attention instead of manifold constraint gradients (ablation) | DDIBs · AIST++ dance | FPD: 0.1208 -> 0.1748 | Hurt |
| incorporate source keyframes through cross-attention instead of manifold constraint gradients (ablation) | DDIBs · AIST++ dance | FMD: 0.1214 -> 0.1313 | No clear effect |
| add keyframe manifold constraint gradients during transfer (ablation) | DDIBs-vanilla · 100STYLE | cycle consistency L2 distance: 0.0198 ± 0.0083 -> 0.0192 ± 0.0079 | No clear effect |
| add keyframe manifold constraint gradients during transfer (ablation) | DDIBs-vanilla · AIST++ | cycle consistency L2 distance: 0.0264 ± 0.0035 -> 0.0232 ± 0.0032 | Helped |
| add keyframe manifold constraint gradients during transfer (ablation) | DDIBs-vanilla · dance user study | motion naturalness: 0.2 | No clear effect |
| add keyframe manifold constraint gradients during transfer (ablation) | DDIBs-vanilla · dance user study | transfer strength: 0.35 | No clear effect |
| add keyframe manifold constraint gradients during transfer (ablation) | DDIBs-vanilla · dance user study | body shape preservation | Helped |
Surprising findings
- DDIBs-gradient outperforms DDIBs-explicit on FPD (0.1208 vs 0.1748), even though explicit keyframe context is given through cross-attention.
Limitations
- Content preservation for limb movements remains poor in both systems according to the human study.
- Evaluation is limited to 100STYLE forward walking and AIST++ dance; generalization beyond these is untested.
- DDIBs struggle when source and target domains are dissimilar, e.g., in dance transfer.
Experimental setup
Datasets: 100STYLE AIST++
Benchmarks: AIST++ 100STYLE
Metrics: Fréchet Pose distance (FPD) Fréchet Motion distance (FMD) L2 cycle consistency distance human preference / Likert ratings McNemar test two one-sided equivalence tests
Baselines: StarDance DDIBs-vanilla DDIBs-explicit CycleDance
Builds on: Dual Diffusion Implicit Bridges EDGE Manifold Constrained Gradient DDIM CycleDance
Closest papers by meaning
- Arbitrary Motion Style Transfer with Multi-condition Motion Latent Diffusion Model (CVPR 2024 · similarity 0.88)
- MotionMix: Weakly-Supervised Diffusion for Controllable Motion Generation (AAAI 2024 · similarity 0.87)
- ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bank (AAAI 2024 · similarity 0.87)
- Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer (CVPR 2024 · similarity 0.87)
- Human Motion Diffusion as a Generative Prior (ICLR 2024 · similarity 0.86)
- Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.