Research map / 3D vision / Human pose estimation
TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TokenHMR, a 3D human mesh recovery method that predicts discrete VQ-VAE pose tokens and uses a threshold-adaptive loss (TALS) to reduce camera/pose bias from 2D keypoints and pseudo-ground-truth.
Problem
Single-image 3D human pose and shape regression suffers a decline in 3D accuracy as 2D keypoint fit improves, due to approximate camera models and biased pseudo-ground-truth (p-GT) used in training.
Approach
Combine a threshold-adaptive loss that down-weights 2D and p-GT errors below thresholds derived from BEDLAM camera-bias analysis with a frozen VQ-VAE pose tokenizer pre-trained on mocap, so pose regression becomes token classification over a codebook of valid poses.
Key results
- On EMDB, TokenHMR reaches 104.2 MVE / 88.1 MPJPE / 49.8 PA-MPJPE with BEDLAM training, versus HMR2.0's 106.6 / 90.7 / 51.3.
- With the full SD+ITW+BL data mix, TokenHMR reports 109.4 MVE / 91.7 MPJPE on EMDB versus 120.7 / 99.3 for HMR2.0, a roughly 7.6% MPJPE reduction.
- Tokenization costs about 2.5mm of 3D accuracy according to the authors, which they describe as negligible relative to SOTA errors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TALS to HMR2.0 (ablation) | HMR2.0 trained on SD + ITW + BL · EMDB | MPJPE: 99.3 -> 96.7 | Helped |
| add TALS to HMR2.0 (ablation) | HMR2.0 trained on SD + ITW + BL · 3DPW | MPJPE: 77.4 -> 73.5 | Helped |
| replace continuous pose regression with tokenized pose prediction (ablation) | HMR2.0 trained on SD + ITW + BL · EMDB | MPJPE: 99.3 -> 95.6 | Helped |
| replace continuous pose regression with tokenized pose prediction (ablation) | HMR2.0 trained on SD + ITW + BL · 3DPW | MPJPE: 77.4 -> 75.0 | Helped |
| use VPoser rather than the token-based prior, while retaining TALS (ablation) | HMR2.0 trained on SD + ITW + BL · EMDB | MPJPE: 91.7 vs 97.9 | Hurt |
| use 80 rather than 160 pose tokens (ablation) | VQ-VAE tokenizer · MOYO validation | MPJPE: 10.4 -> 16.7 | Hurt |
| increase pose tokens from 160 to 320 (ablation) | VQ-VAE tokenizer · MOYO validation | MPJPE: 10.4 -> 10.1 | Helped |
| use codebook 1024 × 256 instead of 2048 × 256 (ablation) | VQ-VAE tokenizer · AMASS test | MPJPE: 2.2 -> 4.6 | Hurt |
| use codebook 2048 × 128 instead of 2048 × 256 (ablation) | VQ-VAE tokenizer · AMASS test | MPJPE: 2.2 -> 3.1 | Hurt |
| add random pose-parameter noise during tokenizer training (ablation) | VQ-VAE tokenizer · MOYO validation | MPJPE: 11.5 -> 10.4 | Helped |
| add random pose-parameter noise during tokenizer training (ablation) | VQ-VAE tokenizer · AMASS test | MPJPE: 1.9 -> 2.2 | Hurt |
| train TokenHMR with tokenized pose prediction and TALS | HMR2.0 trained on SD + ITW + BL · EMDB | MPJPE: 99.3 -> 91.7 | Helped |
Surprising findings
- Adding in-the-wild 2D data to HMR2.0 degraded its EMDB 3D accuracy by over 17%, even though 2D accuracy improved.
- HMR2.0 achieved lower PCK than ground-truth 3D bodies projected with its camera on BEDLAM, indicating camera bias drives 2D-accurate but 3D-wrong outputs.
Limitations
- Discretization of pose into tokens incurs about 2.5mm loss in 3D accuracy.
- Camera model remains approximate; the method mitigates rather than solves camera bias.
- Evaluation focuses on EMDB and 3DPW; other benchmarks are not reported in the main text.
Experimental setup
Datasets: AMASS MOYO Human3.6M MPI-INF-3DHP COCO MPII InstaVariety AVA AI Challenger BEDLAM
Benchmarks: EMDB 3DPW LSP
Metrics: MVE MPJPE PA-MPJPE PCK
Baselines: HybrIK CLIFF HMR2.0 BEDLAM-CLIFF VPoser-augmented HMR2.0 HMR2.0 + TALS
Builds on: HMR2.0 VQ-VAE T2M-GPT CLIFF VPoser
Closest papers by meaning
- GenHMR: Generative Human Mesh Recovery (AAAI 2025 · similarity 0.90)
- FactorizedHMR: A Hybrid Framework for Video Human Mesh Recovery (NeurIPS 2026 · similarity 0.89)
- Self-Supervised 3D Human Mesh Recovery from a Single Image with Uncertainty-Aware Learning (AAAI 2024 · similarity 0.89)
- ManiPose: Manifold-Constrained Multi-Hypothesis 3D Human Pose Estimation (NeurIPS 2024 · similarity 0.88)
- Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimation (CVPR 2024 · similarity 0.88)
- BLADE: Single-view Body Mesh Estimation through Accurate Depth Estimation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.