Research map / 3D vision / Human pose estimation
MEGA: Masked Generative Autoencoder for Human Mesh Recovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MEGA, a masked generative autoencoder over discrete human mesh tokens (from a frozen Mesh-VQ-VAE) that performs HMR in a deterministic single-pass mode or a stochastic iterative mode producing multiple meshes.
Problem
Single-image human mesh recovery is ambiguous due to depth ambiguity; single-output regressors ignore this, and existing multi-output generative methods lose accuracy when producing a single prediction.
Approach
Formulate HMR as generating a sequence of discrete mesh tokens conditioned on image features using a masked generative encoder-decoder Transformer, pre-trained self-supervised on motion capture token sequences, with the encoder discarded at deterministic inference and iterative MaskGIT-style sampling for stochastic inference.
Key results
- Deterministic mode with HRNet-w48 reaches 81.6 PVE / 68.5 MPJPE / 44.1 PA-MPJPE on 3DPW and 107.9 PVE on EMDB, outperforming VQ-HPS (84.8 / 112.9).
- With ViT-H backbone, MEGA achieves 80.0 PVE on 3DPW, beating HMR2.0 (84.1) and TokenHMR (84.6 with extra data).
- In stochastic mode with ResNet-50, MEGA reaches 58.6 PA-MPJPE with one sample vs 59.8 for ProHMR at Q=1 and 47.6 at Q=25, and it needs about 10 samples with HRNet to beat deterministic mode.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cosine-based variable masking instead of linear masking (ablation) | MEGA training · 3DPW | PVE (mm): 86.5 -> 81.6 | Helped |
| Use cosine-based variable masking instead of linear masking (ablation) | MEGA training · EMDB | PVE (mm): 118.7 -> 107.9 | Helped |
| Use variable masking rather than full masking during supervised training (ablation) | MEGA training · 3DPW | PVE (mm): 81.8 -> 81.6 | Helped |
| Use variable masking rather than full masking during supervised training (ablation) | MEGA training · EMDB | PVE (mm): 110.3 -> 107.9 | Helped |
| Add self-supervised pre-training on human meshes (ablation) | MEGA · 3DPW | PVE (mm): 84.1 -> 81.6 | Helped |
| Add self-supervised pre-training on human meshes (ablation) | MEGA · EMDB | PVE (mm): 113.9 -> 107.9 | Helped |
| Use MEGA instead of VQ-HPS | HRNet-w48 HMR · 3DPW | PVE (mm): 84.8 -> 81.6 | Helped |
| Use MEGA instead of VQ-HPS | HRNet-w48 HMR · EMDB | PVE (mm): 112.9 -> 107.9 | Helped |
| Generate multiple stochastic predictions rather than one | MEGA with ResNet-50 · 3DPW | PVE (mm), minimum error over samples: 101.6 -> 87.5 | Helped |
| Generate multiple stochastic predictions rather than one | MEGA with HRNet · 3DPW | PVE (mm), minimum error over samples: 83.4 -> 75.1 | Helped |
Surprising findings
- Discarding the encoder at deterministic inference keeps accuracy while reducing model size.
- Training with 100% masking during supervised training slightly degraded performance versus variable masking, suggesting shared procedures across training stages matter.
Limitations
- Multi-sample stochastic mode needs about 10 samples to beat deterministic mode with HRNet backbone
- Comparisons with TokenHMR are not fully fair due to extra 2D data and BEDLAM training
Experimental setup
Datasets: AMASS MSCOCO Human3.6M MPI-INF-3DHP MPII
Benchmarks: 3DPW EMDB 3DPW-OCC
Metrics: PVE MPJPE PA-MPJPE relative improvement
Baselines: VQ-HPS TokenHMR HMR2.0 CLIFF PARE FastMETRO Virtual Marker Diff-HMR ProHMR 3D Multibodies ScoreHypo SPIN
Builds on: Mesh-VQ-VAE VQ-HPS masked autoencoder MaskGIT HRNet
Closest papers by meaning
- GenHMR: Generative Human Mesh Recovery (AAAI 2025 · similarity 0.92)
- SAT-HMR: Real-Time Multi-Person 3D Mesh Estimation via Scale-Adaptive Tokens (CVPR 2025 · similarity 0.86)
- SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusion (CVPR 2024 · similarity 0.86)
- ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization (ICML 2025 · similarity 0.86)
- A Unified Masked Autoencoder with Patchified Skeletons for Motion Synthesis (AAAI 2024 · similarity 0.86)
- TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.