Research map / Video / Video generation
MotionDreamer: One-to-Many Motion Synthesis with Localized Generative Masked Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MotionDreamer, a localized generative masked transformer with codebook distribution regularization and sliding-window local attention that synthesizes diverse motions from a single reference motion.
Problem
Generating diverse yet faithful motions from a single MoCap reference; generative masked transformers overfit and collapse when trained on one sequence, and prior GAN, diffusion and motion-matching methods trade off expressiveness against diversity and naturalness.
Approach
Shrinks the transformer's receptive field with sliding-window local attention (SlidAttn, overlapping windows, window-wise relative positional encoding, learnable queries, overlap attention fusion) and adds a KL-based codebook distribution regularizer to prevent code collapse when tokenizing a single motion.
Key results
- On SinMotion, MotionDreamer reaches Harmonic Mean 0.43 versus 0.36 for SinMDM and 0.32 for GenMM, with best Global Diversity 1.33 and Inter Diversity 0.25 and Intra Diversity 0.28.
- GenMM attains the highest coverage (95.29%) while MotionDreamer reaches 93.47%.
- Removing the codebook regularization L_token lowers VQ perplexity from 28.13 to 24.56 and harmonic mean from 0.43 to 0.34.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MotionDreamer instead of SinMDM | Single-reference motion synthesis · SinMotion | Harmonic Mean: 0.36 -> 0.43 | Helped |
| Use MotionDreamer instead of GenMM | Single-reference motion synthesis · SinMotion | Global Diversity: 0.50 -> 1.33 | Helped |
| Add codebook distribution regularization (ablation) | MotionDreamer tokenization · SinMotion | Coverage (%): 87.26 -> 93.47 | Helped |
| Add codebook distribution regularization (ablation) | MotionDreamer tokenization · SinMotion | Harmonic Mean: 0.34 -> 0.43 | Helped |
| Replace standard transformer blocks with SlidAttn (ablation) | MotionDreamer · SinMotion | Harmonic Mean: 0.07 -> 0.35 | Helped |
| Add differentiable dequantization to SlidAttn (ablation) | MotionDreamer · SinMotion | Harmonic Mean: 0.35 -> 0.43 | Helped |
| Set codebook size to 32 instead of 48 (ablation) | MotionDreamer · SinMotion | Harmonic Mean: 0.43 -> 0.24 | Hurt |
| Set codebook size to 64 instead of 48 (ablation) | MotionDreamer · SinMotion | Harmonic Mean: 0.43 -> 0.22 | Hurt |
| Replace SlidAttn with Local SASA (ablation) | MotionDreamer · SinMotion | Coverage (%): 93.47 -> 74.23 | Hurt |
| Replace SlidAttn with QnA (ablation) | MotionDreamer · SinMotion | Coverage (%): 85.02 -> 93.47 | Helped |
| Remove learnable queries (ablation) | SlidAttn in MotionDreamer · SinMotion | Harmonic Mean: 0.43 -> 0.23 | Hurt |
| Remove relative positional encoding (ablation) | SlidAttn in MotionDreamer · SinMotion | Harmonic Mean: 0.43 -> 0.22 | Hurt |
Surprising findings
- A standard transformer trained on a single motion overfits badly, with harmonic mean 0.07 and diversity near zero.
- Codebook regularization gives negligible gains for MoMask (R precision unchanged at 0.504), though it helps MMM substantially.
Limitations
- Performance depends heavily on the quality of the single reference motion.
- Limited generalization to broader editing and conditional synthesis tasks due to single-instance setting.
- Cannot capture long-range dependencies and global patterns well.
Experimental setup
Datasets: Mixamo TrueboneZOO SinMotion
Benchmarks: SinMotion
Metrics: Coverage Global Diversity Local Diversity Inter Diversity Intra Diversity diff Harmonic Mean VQ Perplexity R Precision Top 1 user study scores (coverage, diversity, naturalness)
Baselines: Ganimator GenMM SinMDM Local SASA QnA
Builds on: MoMask MMM SinMDM GenMM Ganimator
Closest papers by meaning
- MMM: Generative Masked Motion Model (CVPR 2024 · similarity 0.90)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.88)
- Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation (CVPR 2025 · similarity 0.87)
- MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion (ICLR 2025 · similarity 0.87)
- Make Pixels Dance: High-Dynamic Video Generation (CVPR 2024 · similarity 0.87)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.