Research map / 3D vision / Human pose estimation
PoseGen: Learning to Generate 3D Human Pose Dataset with NeRF
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PoseGen, an end-to-end framework that trains a NeRF-based generator of 3D human poses and camera viewpoints with feedback from a pre-trained pose estimator, producing OOD data that fine-tunes SPIN and HybrIK.
Problem
Pose estimators trained on limited public 3D human pose datasets generalize poorly to out-of-distribution poses and camera viewpoints, and existing augmentation methods either ignore image rendering or do not target the specific pre-trained model.
Approach
A GAN-style pose and viewpoint generator feeds a frozen animatable NeRF (A-NeRF) that renders images; the pre-trained pose estimator's error serves as feedback, and the generator is trained to maximize that error (scenario 2) so the generated data are OOD for the estimator, then the estimator is fine-tuned on this data.
Key results
- PoseGen fine-tuning improves SPIN on 3DPW from 96.9 to 89.7 MPJPE and HybrIK from 88.7 to 84.4 MPJPE, with an average of about 6% relative improvement across four datasets.
- On AGORA, Ours+HybrIK reaches 132.7 MPJPE versus 137.9 for pre-trained HybrIK; on SKI-Pose, Ours+HybrIK reaches 204.2 MPJPE versus 205.2.
- Scenario 2 (maximizing estimator error, OOD generation) gives 91.8 MPJPE on 3DPW versus 92.6 for scenario 1 (IND generation).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune SPIN with PoseGen-generated data | SPIN · AGORA | MPJPE: 175.1 -> 158.9 | Helped |
| Fine-tune HybrIK with PoseGen-generated data | HybrIK · AGORA | MPJPE: 137.9 -> 132.7 | Helped |
| Fine-tune SPIN with PoseGen-generated data | SPIN · 3DPW | MPJPE: 96.9 -> 89.7 | Helped |
| Fine-tune HybrIK with PoseGen-generated data | HybrIK · 3DPW | MPJPE: 88.7 -> 84.4 | Helped |
| Fine-tune SPIN with PoseGen-generated data | SPIN · SKI-Pose | MPJPE: 288.9 -> 250.9 | Helped |
| Fine-tune HybrIK with PoseGen-generated data | HybrIK · SKI-Pose | MPJPE: 205.2 -> 204.2 | Helped |
| Fine-tune HybrIK with PoseGen-generated data | HybrIK · 3DHP | MPJPE: 99.3 -> 96.7 | Helped |
| Add novel pose generation (ablation) | PoseGen with SPIN on 3DPW · 3DPW | MPJPE: 96.9 -> 95.8 | Helped |
| Add camera-viewpoint generation to novel pose generation (ablation) | PoseGen with SPIN on 3DPW · 3DPW | MPJPE: 95.8 -> 92.7 | Helped |
| Add feedback to pose and viewpoint generation (ablation) | PoseGen with SPIN on 3DPW · 3DPW | MPJPE: 92.7 -> 91.3 | Helped |
| Generate OOD rather than IND data (Scenario 2 rather than Scenario 1) (ablation) | PoseGen with SPIN · 3DPW | MPJPE: 92.6 -> 91.8 | Helped |
| Use a normal rather than uniform latent prior (ablation) | PoseGen with SPIN · 3DPW | MPJPE: 91.9 -> 91.3 | Helped |
Surprising findings
- Normal latent prior outperformed uniform and spherical priors, contrary to prior pose-prior work favoring uniform-like distributions.
- Generating only 6K samples was enough to reach the best performance, with gains saturating beyond that.
Limitations
- A-NeRF renders only a single subject, so a separate NeRF checkpoint is needed per subject.
- NeRF rendering fails on complex poses and novel viewpoints, producing some erroneous samples.
- Gains are modest on some benchmarks (e.g., 0.9 mm over CLIFF on 3DPW, and SPIN is worse on AGORA-style settings).
Experimental setup
Datasets: AMASS 3DPW AGORA SKI-Pose MPI-INF-3DHP Human3.6M MSCOCO LSP SURREAL
Benchmarks: 3DPW AGORA SKI-Pose MPI-INF-3DHP USD (user-specific dataset)
Metrics: MPJPE PA-MPJPE NMJE PCK
Baselines: SPIN HybrIK PoseAug AGORA BEDLAM CLIFF VIBE EFT
Builds on: A-NeRF SPIN HybrIK PoseAug AdaptPose
Closest papers by meaning
- A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (NeurIPS 2021 · similarity 0.91)
- AdaptPose: Cross-Dataset Adaptation for 3D Human Pose Estimation by Learnable Motion Generation (CVPR 2022 · similarity 0.88)
- FlexPose: Pose Distribution Adaptation with Limited Guidance (AAAI 2025 · similarity 0.88)
- ICON: Incremental CONfidence for Joint Pose and Radiance Field Optimization (CVPR 2024 · similarity 0.87)
- A Dual-Augmentor Framework for Domain Generalization in 3D Human Pose Estimation (CVPR 2024 · similarity 0.86)
- GenHMR: Generative Human Mesh Recovery (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.