Research map / Video / Video segmentation and tracking
MoGenTS: Motion Generation based on Spatial-Temporal Joint Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Quantizes each human joint (across time) into its own VQ token to form a spatial-temporal 2D token map, then generates motion with masked 2D transformer using temporal-spatial masking and spatial-temporal attention.
Problem
Text-to-motion generation from discrete quantization suffers from approximation errors, and whole-body single-code quantization both overloads each code and loses inter-joint spatial relationships.
Approach
Quantize each joint at each frame into its own code, yielding a 2D (joint x time) token map that enables 2D convolution, 2D positional encoding, and 2D attention; pair this with temporal-then-spatial 2D masking and joint-spatial/joint-temporal attention in a masked transformer.
Key results
- On HumanML3D, FID 0.033 vs. 0.045 for MoMask (26.6% decrease from prior SOTA) and Top-1 0.529.
- On KIT-ML, FID 0.143 vs. 0.204 for MoMask (29.9% decrease), Top-1 0.445.
- Motion quantization on HumanML3D reduces MPJPE to 13.8 mm vs. 29.5 mm for MoMask's VQ-VAE at the same 512x512 codebook size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MoGenTS instead of MoMask | Text-to-motion generation · HumanML3D | FID: 0.045^{± .002} -> 0.033^{± .001} | Helped |
| use MoGenTS instead of MoMask | Text-to-motion generation · KIT-ML | FID: 0.204^{± .011} -> 0.143^{± .004} | Helped |
| replace 1D motion VQVAE with 2D joint VQVAE (ablation) | HumanML3D text-to-motion baseline · HumanML3D | FID: 0.108 -> 0.194 | Hurt |
| add temporal-spatial 2D token masking (ablation) | HumanML3D model with 2D VQ · HumanML3D | FID: 0.194 -> 0.054 | Helped |
| add 2D position encoding (ablation) | HumanML3D model with 2D masking · HumanML3D | FID: 0.054 -> 0.047 | Helped |
| add joint spatial and temporal attention (ablation) | HumanML3D model with 2D position encoding · HumanML3D | FID: 0.047 -> 0.033 | Helped |
Surprising findings
- Naively switching from 1D VQ to joint-level 2D VQ hurt generation (FID rose from 0.108 to 0.194) despite better reconstruction, attributed to a large increase in token count.
- Simple random spatial masking outperformed graph-based masking strategies.
Limitations
- Approximation error from quantization still limits generation quality.
- Restricted size of current human motion datasets limits quantizer quality versus image pretraining.
- Inference is slower than MoMask (181 ms vs 73 ms).
Experimental setup
Datasets: HumanML3D KIT-ML Motion-X
Benchmarks: HumanML3D KIT-ML Motion-X
Metrics: FID R-precision (Top-1/2/3) MM-Dist Diversity MPJPE
Baselines: TEMOS TM2T T2M MDM MLD MotionDiffuse PhysDiff MotionGPT T2M-GPT M2DM Fg-T2M AttT2M
Builds on: MoMask T2M-GPT BERT masked modeling residual VQ-VAE classifier-free guidance
Closest papers by meaning
- HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation (CVPR 2025 · similarity 0.86)
- Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregression (CVPR 2025 · similarity 0.86)
- Shape My Moves: Text-Driven Shape-Aware Synthesis of Human Motions (CVPR 2025 · similarity 0.85)
- VideoMDM: Towards 3D Human Motion Generation From 2D Supervision (NeurIPS 2026 · similarity 0.85)
- SAMoR: Motion Modelling for Articulated Objects of Any Skeleton and Topology (NeurIPS 2026 · similarity 0.85)
- Video-Based Human Pose Regression via Decoupled Space-Time Aggregation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.