Research map / 3D vision / Human pose estimation
PoseBH: Prototypical Multi-Dataset Training Beyond Human Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PoseBH, a multi-dataset pose estimation framework using nonparametric keypoint prototypes in a shared embedding space plus cross-type self-supervision, improving whole-body and animal pose estimation while preserving human pose accuracy.
Problem
Multi-dataset training for pose estimation is hard because skeletons differ across datasets (heterogeneous keypoint definitions) and each image has only one skeleton annotation, leaving other skeleton types unsupervised.
Approach
Represent keypoints as learnable-free (nonparametric, online-clustered) prototypes in a unified embedding space so keypoints from different skeletons are matched automatically, and supervise unlabeled skeleton types by fusing keypoint-head and prototype-head predictions (cross-type self-supervision) without teacher networks or extra augmentation.
Key results
- Average over six MDT datasets improves from 68.2 (multi-head baseline) to 70.6 with prototypes plus cross-type self-supervision (ablation, ViT-B).
- On animal/whole-body benchmarks, achieves 73.4 average AP vs. 69.1 for ViTPose++, including 11.2 AP gain on APT-36K.
- With ViT-H backbone reaches 79.5 AP on COCO validation, state-of-the-art among compared methods; transfers to 3DPW with 83.6 AP vs 81.7 for ViTPose++.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PoseBH multi-dataset training instead of ViTPose++ | ViT-B pose estimation · AP-10K | AP: 74.1 -> 75.0 | Helped |
| use PoseBH multi-dataset training instead of ViTPose++ | ViT-B pose estimation · APT-36K | AP: 76.0 -> 87.2 | Helped |
| use PoseBH multi-dataset training instead of ViTPose++ | ViT-B pose estimation · COCO-WholeBody | AP: 57.1 -> 57.9 | Helped |
| use PoseBH multi-dataset training instead of ViTPose++ | ViT-B pose estimation · COCO | AP: 77.0 -> 77.3 | Helped |
| use PoseBH multi-dataset training instead of UniHCP | ViT-B pose estimation · AIC | AP: 32.6 -> 32.1 | Hurt |
| use PoseBH multi-dataset training instead of ViTPose++ | ViT-B pose estimation · MPII | PCK: 93.1 -> 93.2 | No clear effect |
| transfer PoseBH representations instead of ViTPose++ | ViT-B model fine-tuned for InterHand2.6M · InterHand2.6M | PCK: 98.3 -> 98.6 | No clear effect |
| transfer PoseBH representations instead of ViTPose++ | ViT-B model fine-tuned for 3DPW · 3DPW | AP: 81.7 -> 83.6 | Helped |
| add keypoint prototype learning (ablation) | PoseBH multi-dataset training · COCO, AIC, MPII, AP-10K, APT-36K, COCO-WholeBody | average score: 68.2 -> 70.4 | Helped |
| add cross-type self-supervision (ablation) | PoseBH with keypoint prototype learning · COCO, AIC, MPII, AP-10K, APT-36K, COCO-WholeBody | average score: 70.4 -> 70.6 | Helped |
Surprising findings
- Naively merging identically named keypoints across datasets is ineffective due to domain-specific localization differences.
Limitations
- Requires at least few-shot samples to learn prototypes, so zero-shot inference on unseen skeletons is infeasible.
- Evaluated only on 2D pose and transfer tasks; 3D adaptation left to future work.
Experimental setup
Datasets: COCO AIChallenger MPII AP-10K APT-36K COCO-WholeBody InterHand2.6M 3DPW
Benchmarks: COCO AIC MPII AP-10K APT-36K COCO-WholeBody InterHand2.6M 3DPW
Metrics: AP AP50 AP75 AR PCKh PCKh@0.1 PCK AUC EPE
Baselines: ViTPose++ UniHCP HRNet HRFormer SimCC PCT
Builds on: ViTPose++ ProtoSeg SimCC JointMSE
Closest papers by meaning
- FlexPose: Pose Distribution Adaptation with Limited Guidance (AAAI 2025 · similarity 0.89)
- Self-Supervised 3D Human Pose Estimation via Part Guided Novel Image Synthesis (CVPR 2020 · similarity 0.88)
- AdaptPose: Cross-Dataset Adaptation for 3D Human Pose Estimation by Learnable Motion Generation (CVPR 2022 · similarity 0.88)
- PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation (CVPR 2021 · similarity 0.88)
- ManiPose: Manifold-Constrained Multi-Hypothesis 3D Human Pose Estimation (NeurIPS 2024 · similarity 0.88)
- Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.