Research map / 3D vision / Human pose estimation
Harmony4D: A Video Dataset for In-The-Wild Close Human Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Harmony4D, a large in-the-wild multi-view video dataset of close human-human contact (wrestling, dancing, MMA, etc.) with markerless 3D pose/mesh annotations, and shows fine-tuning HMR2.0 on it reduces error under occlusion and contact.
Problem
Existing multi-human 3D datasets for close physical interaction are captured in controlled indoor settings with choreographed actions, limiting diversity and generalization; accurate mesh ground truth under severe occlusion and contact is hard to obtain.
Approach
A markerless multi-view annotation pipeline that alternates 3D pose forecasting (per-keypoint Kalman filters), point-prompted instance segmentation with SAM, mask-conditioned 2D pose estimation (SegPose2D), RANSAC triangulation, and SMPL fitting with a mesh interpenetration loss to obtain contact-aware ground-truth meshes from a mobile ~20-camera rig.
Key results
- Fine-tuned HMR2.0 reduces test MPJPE from 108.18 mm to 47.71 mm and PVE from 131.00 to 59.14 mm, a 54.8% PVE improvement.
- Off-the-shelf Multi-HMR is the strongest baseline at 93.82 mm MPJPE, outperforming BUDDI (126.35 mm) which is designed for human contact.
- Interpenetration loss reduces average mP2S by 44.7 mm across hugging, wrestling, dancing, karate and MMA sequences.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune HMR2.0 on Harmony4D train | HMR2.0 · Harmony4D test | MPJPE: 108.18 -> 47.71; improves MPJPE by 55.9% | Helped |
| Fine-tune HMR2.0 on Harmony4D train | HMR2.0 · Harmony4D test | PVE: 131.00 -> 59.14; MPVPE by 54.8% | Helped |
| Fine-tune HMR2.0 on Harmony4D train | HMR2.0 · Harmony4D test | 3DPCK: 75.62 -> 96.63; 3DPCK by 21.01% | Helped |
| Add interpenetration loss to mesh optimization (ablation) | Harmony4D mesh optimization · Hugging, Wrestling, Dancing, Karate, MMA | mP2S: reduced by 44.7 mm on average | Helped |
| Add interpenetration loss to mesh optimization (ablation) | Harmony4D mesh optimization · Hugging, Wrestling, Dancing, Karate, MMA | mIoU: reduced by 0.7% on average | Helped |
| Add interpenetration loss to mesh optimization (ablation) | Harmony4D mesh optimization · Hugging, Wrestling, Dancing, Karate, MMA | mIoA: reduced by 0.1 m^2 on average | Helped |
| Increase the number of cameras used for triangulation (ablation) | Harmony4D pose processing · Harmony4D activities | MPJPE | Helped |
Surprising findings
- Off-the-shelf Multi-HMR outperforms BUDDI, a method specifically designed to model human-human contact.
- Fine-tuning HMR2.0 on Harmony4D improves performance without explicit contact-vertex supervision, with inter-contact relationships preserved in predictions.
Limitations
- Trades off accuracy of 3D pose and shape under contact in favor of in-the-wild capture compared to high-precision indoor systems.
- Very high normalized per-vertex error for all baselines due to occlusion-induced inconsistent meshes.
- Manual inspection and re-optimization of 3D poses and meshes is still required.
Experimental setup
Datasets: Harmony4D COCO 3DPW
Benchmarks: Harmony4D test set 3DPW
Metrics: MPJPE PA-MPJPE N-MPJPE PVE PA-PVE N-PVE 3DPCK AUC F1 mP2S
Baselines: PARE ROMP BEV BUDDI HMR2.0 Multi-HMR
Builds on: EgoHumans Segment Anything ViTPose HMR2.0 SMPL
Closest papers by meaning
- CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangement (CVPR 2025 · similarity 0.88)
- HOI-M^3: Capture Multiple Humans and Objects Interaction within Contextual Environment (CVPR 2024 · similarity 0.88)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.88)
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (CVPR 2024 · similarity 0.87)
- AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation (ICLR 2025 · similarity 0.86)
- Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.