Research map / 3D vision / Human pose estimation
A Unified Masked Autoencoder with Patchified Skeletons for Motion Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UNIMASK-M, a task-agnostic masked autoencoder that decomposes human poses into body-part patches and reformulates motion forecasting, inbetweening, and completion as masked reconstruction.
Problem
Human motion synthesis is typically handled by task-specific models (forecasting, inbetweening, completion, reconstruction) that do not transfer across pose-conditioning patterns and are often generative and slow.
Approach
Decompose each pose into limb/trunk patches (ViT-style) and treat all pose-conditioned motion synthesis tasks as masked reconstruction with a deterministic encoder-decoder, using kinematic embeddings and a delta-from-interpolation output strategy.
Key results
- On LaFAN1 inbetweening, UNIMASK-M reaches L2Q 1.16 and L2P 1.80 at 70 transition frames, versus 1.20 and 1.91 for Delta-Interpolation when both are trained for long transitions.
- On Human3.6M forecasting at 1000 ms, UNIMASK-M achieves 112.1 MPJPE, versus 110.0 for ST-DGCN (best), while being competitive with autoregressive baselines.
- Under 20% joint occlusion on Human3.6M, UNIMASK-M reports 120.5 MPJPE at 1000 ms, versus 124.6 for the best baseline (ST-DGCN).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use UNIMASK-M rather than ST-DGCN | Human3.6M forecasting · Human3.6M forecasting, 1000 ms | MPJPE: 110.0 -> 112.1 | Hurt |
| use UNIMASK-M rather than ST-DGCN | Human3.6M forecasting with occluded input · Human3.6M forecasting with 20% joint occlusion, 1000 ms | MPJPE: 124.6 -> 120.5 | Helped |
| use UNIMASK-M rather than Δ-Interpolation | LaFAN1 inbetweening · LaFAN1, 30 transition frames | L2Q: 0.57 -> 0.57 | No clear effect |
| use UNIMASK-M rather than Δ-Interpolation | LaFAN1 inbetweening · LaFAN1, 70 transition frames | L2Q: 1.20 -> 1.16 | Helped |
| use UNIMASK-M rather than Δ-Interpolation | LaFAN1 inbetweening · LaFAN1, 70 transition frames | L2P: 1.91 -> 1.80 | Helped |
| use UNIMASK-M rather than Δ-Interpolation | LaFAN1 inbetweening · LaFAN1, 70 transition frames | NPSS: 1.0180 -> 0.9532 | Helped |
| add pose decomposition (PD) (ablation) | UNIMASK-M · LaFAN1, 30 transition frames | L2Q: 0.59 -> 0.57 | Helped |
| add pose aggregation (PA) (ablation) | UNIMASK-M · LaFAN1, 30 transition frames | L2P: 1.05 -> 0.97 | Helped |
| add DCT (ablation) | UNIMASK-M · LaFAN1, 30 transition frames | L2Q: 0.57 -> 0.68 | Hurt |
| add TempMLP (ablation) | UNIMASK-M · LaFAN1, 30 transition frames | L2P: 0.97 -> 1.04 | Hurt |
| add kinematic embedding (emb_kin) (ablation) | UNIMASK-M · LaFAN1, 30 transition frames | L2Q: 0.58 -> 0.57 | No clear effect |
| remove the encoder (ablation) | UNIMASK-M · LaFAN1, 30 transition frames | L2Q: 0.57 -> 0.58 | No clear effect |
Surprising findings
- Curriculum masking-probability training gave a unified model performance comparable to task-specific models, and pretraining then finetuning was worse (+0.89 MPJPE on average).
- DCT encoding substantially degraded inbetweening performance on LaFAN1 because its motions are less periodic.
Limitations
- Does not beat state-of-the-art on all forecasting horizons (e.g., ST-DGCN and SiMLPe are better at short horizons on Human3.6M).
- Requires task-specific tuning (DCT and TempMLP only used for Human3.6M) and the unified model underperforms task-specific models at some horizons.
- Motion reconstruction/occlusion evaluation relies on randomly masked joints rather than real occlusion data.
Experimental setup
Datasets: Human3.6M LaFAN1
Benchmarks: Human3.6M LaFAN1
Metrics: MPJPE L2P L2Q NPSS
Baselines: Delta-Interpolation SiMLPe ST-DGCN DCT-GCN mao2020history MotionMixer MSR-GCN SSMCT TG U-CVAE MDM Zero velocity
Builds on: Masked Autoencoders (MAE) Vision Transformer (ViT) CrossViT-style delta strategy (Delta-Interpolation) SiMLPe
Closest papers by meaning
- MMM: Generative Masked Motion Model (CVPR 2024 · similarity 0.86)
- MEGA: Masked Generative Autoencoder for Human Mesh Recovery (CVPR 2025 · similarity 0.86)
- SurMo: Surface-based 4D Motion Modeling for Dynamic Human Rendering (CVPR 2024 · similarity 0.86)
- MoMask: Generative Masked Modeling of 3D Human Motions (CVPR 2024 · similarity 0.85)
- SAMoR: Motion Modelling for Articulated Objects of Any Skeleton and Topology (NeurIPS 2026 · similarity 0.85)
- MotionDreamer: One-to-Many Motion Synthesis with Localized Generative Masked Transformer (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.