Research map / Video / Video generation
SnapMoGen: Human Motion Generation from Expressive Texts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SnapMoGen, a 20K-clip motion-capture dataset with 122K expressive (avg 48-word) captions, and MoMask++, a multi-scale residual VQ model that generates all token scales with one masked transformer.
Problem
Text-to-motion generation is limited to short, general prompts because existing motion-text datasets have brief annotations, which hurts fine-grained control and generalization; existing residual VQ also wastes token capacity and needs multiple generators.
Approach
Segment long mocap sequences into 4-12 s clips with multiple expressive human-written captions (augmented by an LLM), and quantize motion at progressively finer temporal scales with a shared codebook so a single generative masked transformer predicts all tokens conditioned on text.
Key results
- On SnapMoGen, MoMask++ (cross-attention) reaches FID 15.06 and CLIP score 0.685 vs. MoMask's 17.404 FID and 0.664 CLIP score.
- On HumanML3D, MoMask++ (in-context) achieves R-Precision Top-1 0.528 and FID 0.072, slightly above MoMask's 0.521 Top-1 and 0.045 FID in FID terms, with best Top-1/Top-3 R-precision.
- MoMask++ uses 2 VQ layers on SnapMoGen (a quarter of MoMask's token count) and outperforms MoMask with similar model size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-scale rather than full-scale quantization (ablation) | MoMask++ · SnapMoGen test | T2M FID: 18.02 -> 15.94 | Helped |
| Add self-attention layers to the VQ encoder and decoder (ablation) | MoMask++ · SnapMoGen test | T2M FID: 16.18 -> 15.94 | Helped |
| Use 148 rather than 296 pose features (ablation) | MoMask++ · SnapMoGen test | T2M FID: 15.94 -> 15.95 | No clear effect |
| Reduce codebook size from 2048 to 1024 (ablation) | MoMask++ · SnapMoGen test | T2M FID: 15.94 -> 15.61 | Helped |
| Reduce codebook size from 2048 to 512 (ablation) | MoMask++ · SnapMoGen test | T2M FID: 15.94 -> 16.96 | Hurt |
| Vary the number of quantization layers (ablation) | MoMask++ · SnapMoGen test | T2M FID: 5 layers: 16.38; 3 layers: 16.21; 2 layers: 15.56; 1 layer: 16.25 | Mixed |
| Add text augmentation (ablation) | MoMask++ · SnapMoGen test | T2M FID: 17.98 -> 15.56 | Helped |
| Replace T5-base with a CLIP text encoder (ablation) | MoMask++ · SnapMoGen test | CLIP Score: 0.684 -> 0.478 | Hurt |
| Replace in-context conditioning with cross-attention (ablation) | MoMask++ · SnapMoGen test | FID: 15.56 -> 15.06 | Helped |
| Increase transformer size to the medium configuration (ablation) | MoMask++ · SnapMoGen test | FID: 15.06 -> 15.58 | Hurt |
| Increase transformer size to the large configuration (ablation) | MoMask++ · SnapMoGen test | FID: 15.06 -> 16.02 | Hurt |
Surprising findings
- Larger transformers (medium/large) and higher latent dimensions degraded generation quality, indicating overfitting on SnapMoGen.
- The in-context MoMask++ variant performs better on HumanML3D but overfits on long SnapMoGen prompts, where cross-attention is better.
Limitations
- Inertial mocap suit causes global position imprecision and jitter; cannot capture highly skilled motions like cartwheels or outdoor activities.
- VQ quantization errors degrade motion quality; struggles with rare motions and uncommon prompts.
- Does not maintain physical plausibility such as proper foot contacts.
Experimental setup
Datasets: SnapMoGen HumanML3D BABEL KIT-ML Motion-X
Benchmarks: HumanML3D SnapMoGen
Metrics: R-Precision FID MultiModal Distance Multimodality CLIP score VQ reconstruction FID joint position error
Baselines: TM2T T2M MDM MLD MotionDiffuse T2M-GPT MMM MoMask StableMoFusion MARDM
Builds on: MoMask T2M-GPT TMR HumanML3D
Closest papers by meaning
- MMM: Generative Masked Motion Model (CVPR 2024 · similarity 0.90)
- OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers (CVPR 2024 · similarity 0.90)
- ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models (AAAI 2025 · similarity 0.89)
- AMD: Autoregressive Motion Diffusion (AAAI 2024 · similarity 0.89)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.88)
- MoMask: Generative Masked Modeling of 3D Human Motions (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.