Research map / 3D vision / Human pose estimation
HeatFormer: A Neural Optimizer for Multiview Human Mesh Recovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HeatFormer, a Transformer-based neural optimizer that iteratively refines SMPL parameters for multiview human mesh recovery by aligning joint heatmaps of view-dependent estimates with input multiview heatmaps.
Problem
Multiview human mesh (SMPL shape and pose) recovery from static fixed cameras that is robust to occlusion and agnostic to the number and configuration of cameras.
Approach
Recast SMPL fitting as heatmap generation and alignment: a HeatEncoder consolidates per-joint heatmaps into one heatmap per view, and a decoder cross-attends between input heatmap/image features and heatmaps rendered from the current SMPL estimate, unrolled for ~3 iterations to act as a learned optimizer.
Key results
- On Human3.6M, HeatFormer (iter4) reaches 29.5 MPJPE and 22.4 PA-MPJPE, the best among the compared published methods, versus U-HMR at 31.0 and 22.8.
- On MPI-INF-3DHP, HeatFormer (iter3) achieves 39.8 MPJPE, 99.5 PCK and 72.8 AUC, outperforming PaFF (48.4 MPJPE, 67.3 AUC).
- In cross-dataset evaluation (train Human3.6M, test MPI-INF-3DHP), HeatFormer reaches 56.0 MPJPE versus 73.2 for U-HMR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase decoder inference from one to three iterations (ablation) | HeatFormer · Human3.6M | MPJPE: 34.9 -> 30.7 | Helped |
| increase decoder inference from one to three iterations (ablation) | HeatFormer · MPI-INF-3DHP | MPJPE: 44.6 -> 39.8 | Helped |
| increase views used for training and testing from one to four (ablation) | HeatFormer · Human3.6M | MPJPE: 47.1 -> 29.3 | Helped |
| use four rather than three decoder inference iterations (ablation) | HeatFormer · Human3.6M | MPJPE: 30.7 -> 29.5 | Helped |
| use four rather than three decoder inference iterations (ablation) | HeatFormer · MPI-INF-3DHP | MPJPE: 39.8 -> 40.6 | Hurt |
| use four rather than three decoder inference iterations (ablation) | HeatFormer · BEHAVE | MPJPE: 51.1 -> 48.9; 34.2 -> 32.6 | Helped |
| use multiview HeatFormer rather than U-HMR | Human3.6M · Human3.6M | MPJPE: 31.0 -> 30.7 | Helped |
| use HeatFormer rather than PaFF | MPI-INF-3DHP · MPI-INF-3DHP | MPJPE: 48.4 -> 39.8 | Helped |
| use HeatFormer rather than U-HMR | MPI-INF-3DHP trained on Human3.6M · MPI-INF-3DHP, trained on Human3.6M | MPJPE: 73.2 -> 56.0 | Helped |
| use HeatFormer rather than HMR2.0a | BEHAVE · BEHAVE Protocol 1 | MPJPE: 72.2 -> 51.1 | Helped |
| use HeatFormer rather than HMR2.0a+scoreHMR | BEHAVE · BEHAVE Protocol 1 | PA-MPJPE: 30.9 -> 32.2 | Hurt |
| use HeatFormer rather than HMR2.0a | BEHAVE · BEHAVE Protocol 2 | MPJPE: 48.1 -> 34.2 | Helped |
Limitations
- Relies on AdaFuse multiview joint heatmaps as input, so quality depends on that detector.
- Evaluated on Human3.6M/MPI-INF-3DHP/BEHAVE; temporal information left for future work.
- Global orientation requires a separate pre-trained Transformer; translation needs calibration or weak-perspective estimation.
Experimental setup
Datasets: Human3.6M MPI-INF-3DHP BEHAVE COCO MPII
Benchmarks: Human3.6M MPI-INF-3DHP BEHAVE
Metrics: MPJPE PA-MPJPE MPVPE PCK AUC
Baselines: HMR2.0 LVS PaFF U-HMR ProHMR ScoreHMR PostureHMR Shape-Aware MV-SPIN
Builds on: HMR SPIN AdaFuse HMR2.0 U-HMR
Closest papers by meaning
- MEGA: Masked Generative Autoencoder for Human Mesh Recovery (CVPR 2025 · similarity 0.85)
- ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization (ICML 2025 · similarity 0.85)
- GenHMR: Generative Human Mesh Recovery (AAAI 2025 · similarity 0.85)
- PostureHMR: Posture Transformation for 3D Human Mesh Recovery (CVPR 2024 · similarity 0.84)
- MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware Diffusion and Iterative Refinement (AAAI 2025 · similarity 0.84)
- Incorporating Test-Time Optimization into Training with Dual Networks for Human Mesh Recovery (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.