Research map / 3D vision / Human pose estimation
Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Neural Localizer Fields, a hypernetwork-based continuous field of 3D heatmap point localizers that can query any canonical body point, plus a fast Kabsch-based SMPL-family fitting step, trained on a mixture of heterogeneous annotated human datasets.
Problem
Single-image 3D human pose and shape estimation models are tied to fixed joint/keypoint sets and parametric formats, so heterogeneous annotations (meshes, skeletons, 2D keypoints, DensePose) must be converted, which is costly and error-prone.
Approach
A neural field maps any canonical 3D body point to the weights of a 1x1 convolutional heatmap localizer (hypernetwork), so one model can localize arbitrary points supervised by any annotation type; a fast alternating Kabsch/linear-least-squares solver then fits SMPL-family models to the nonparametric outputs.
Key results
- On EMDB, NLF-L +fit reaches 68.4 mm MPJPE versus 98.0 mm for the second-best method (HMR 2.0 / BEDLAM-CLIFF).
- On SSP-3D, NLF-S +fit achieves 10.0 mm PVE-T-SC and 0.85 mIoU, improving over ShapeBoost (11.4 mm); multi-image shared-beta gives 9.6 mm.
- The Kabsch-based fitter takes 28.4 ms versus 33 minutes for the official SMPL-to-SMPL-X transfer code, at 7.8 mm vs 5.0 mm error.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GPS positional encoding instead of plain XYZ coordinates (ablation) | NLF · SSP-3D | mIoU: 0.777 -> 0.789 | Helped |
| use learnable Fourier features instead of GPS (ablation) | NLF · SSP-3D | mIoU: 0.789 -> 0.804 | Helped |
| initialize learnable Fourier features with GPS (ablation) | NLF · SSP-3D | mIoU: 0.804 -> 0.812 | Helped |
| train with all datasets rather than real datasets only (ablation) | NLF-S · SSP-3D | PVE-T-SC: 11.9 -> 9.9 | Helped |
| train with all datasets rather than synthetic datasets only (ablation) | NLF-S · Human3.6M | MPJPE: 48.0 -> 41.6 | Helped |
| use NLL uncertainty loss instead of Euclidean loss (ablation) | NLF-S nonparam. · 3DPW | MPJPE: 59.2 -> 61.2 | Hurt |
| use beta-NLL instead of NLL uncertainty loss (ablation) | NLF-S nonparam. · 3DPW | PCC-Joints: 0.50 -> 0.54 | Helped |
| use beta-NLL instead of NLL uncertainty loss (ablation) | NLF-S nonparam. · 3DPW | MPJPE: 61.2 -> 59.4 | Helped |
| weight body-model fitting points by estimated uncertainty (ablation) | NLF-L · 3DPW test | MPJPE: 59.3 -> 59.0 | Helped |
| weight body-model fitting points by estimated uncertainty (ablation) | NLF-L · EMDB1 | MPJPE: 68.9 -> 68.4 | Helped |
| fit SMPL parameters to nonparametric predictions (ablation) | NLF-L · EMDB1 | MPJPE: 69.6 -> 68.4 | Helped |
| share shape parameters across multiple images (ablation) | NLF-S +fit · SSP-3D | PVE-T-SC: 10.0 -> 9.6 | Helped |
Surprising findings
- Naively optimizing the NLL uncertainty loss significantly harms mean localization quality; beta-NLL mitigates this.
- Separate per-point convolution weights without a localizer field yield visibly inconsistent skeleton and mesh predictions.
Limitations
- Works frame-by-frame without temporal cues and estimates each person independently, so heavily overlapping people remain hard.
- Predictions may have self-intersections and absolute distance may be inaccurate due to depth/scale ambiguity.
Experimental setup
Datasets: AGORA BEDLAM RICH MOYO ARCTIC InterCap GeneBody EgoBody SPEC Hi4D HuMMan SynBody-NeRF
Benchmarks: 3DPW EMDB AGORA EHF SSP-3D Human3.6M MPI-INF-3DHP MuPoTS-3D
Metrics: MPJPE P-MPJPE MVE P-MVE MPJAE NMVE NMJE PVE-T-SC mIoU PCK
Baselines: PyMAF SMPLer-X BEDLAM-CLIFF HybrIK HMR 2.0 Multi-HMR WHAM-B OSX PyMAF-X MeTRAbs-ACAE
Builds on: MeTRAbs SMPL DensePose EfficientNetV2 beta-NLL
Closest papers by meaning
- Locally Aware Piecewise Transformation Fields for 3D Human Mesh Registration (CVPR 2021 · similarity 0.90)
- Convolutional Mesh Regression for Single-Image Human Shape Reconstruction (CVPR 2019 · similarity 0.90)
- 3D Human Pose Estimation From a Single Image via Distance Matrix Regression (CVPR 2017 · similarity 0.90)
- Learning to Estimate 3D Human Pose and Shape From a Single Color Image (CVPR 2018 · similarity 0.89)
- Coarse-To-Fine Volumetric Prediction for Single-Image 3D Human Pose (CVPR 2017 · similarity 0.89)
- SelfPose3d: Self-Supervised Multi-Person Multi-View 3d Pose Estimation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.