Research map / Label-efficient and robust learning / Recognition and long-tail
Learning Dense Hand Contact Estimation from Imbalanced Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HACO, a ViT-H/HaMeR-based dense hand contact estimator trained on 14 hand-interaction datasets with balanced contact sampling and a vertex-level class-balanced loss to handle class and spatial imbalance.
Problem
Dense hand contact estimation from images is hard to learn because hand contact datasets are dominated by non-contact vertices and contact is concentrated on fingertips.
Approach
Balanced contact sampling groups hand instances by a contact balance score with log-spaced bins and resamples them evenly; a vertex-level class-balanced loss reweights each MANO vertex's BCE by its dataset-wide contact frequency, combined with smoothness and regularization losses.
Key results
- On MOW, HACO reaches F1 0.522 vs. DECO 0.197 and BSTRO 0.112 for dense hand contact.
- On RICH, HACO reaches F1 0.781 vs. BSTRO 0.455 (hand version).
- On DexYCB grasp optimization, HACO improves F1 to 0.666 vs. DeepContact 0.612 and lowers MPJPE to 36.520 from 37.155.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use balanced contact sampling instead of no sampling (ablation) | HACO · MOW | F1-Score: 0.481 -> 0.522 | Helped |
| use vertex-level class-balanced loss instead of CB loss (ablation) | HACO · MOW | F1-Score: 0.465 -> 0.522 | Helped |
| add HO training data to HS+HH+HB (ablation) | HACO · MOW | F1-Score: 0.408 -> 0.522 | Helped |
| add HH training data to HS (ablation) | HACO · MOW | F1-Score: 0.411 -> 0.373 | Hurt |
| add HH training data to HO (ablation) | HACO · RICH | F1-Score: 0.492 -> 0.471 | Hurt |
| add HO training data to HS (ablation) | HACO · Hi4D | F1-Score: 0.419 -> 0.496 | Helped |
| use learned contact initialization instead of no initialization (ablation) | HACO · MOW | F1-Score: 0.503 -> 0.522 | Helped |
| expand training from 1 dataset to 3 datasets (ablation) | HACO · MOW | F1-Score: 0.373 -> 0.485 | Helped |
| expand training from 3 datasets to 14 datasets (ablation) | HACO · MOW | F1-Score: 0.485 -> 0.522 | Helped |
| replace ViT-H with ViT-B backbone (ablation) | HACO · MOW | F1-Score: 0.522 -> 0.484 | Hurt |
Surprising findings
- Plain CB loss yields higher recall but lower F1 than the other losses tested, while several loss variants (e.g., LDAM) trade recall heavily for precision.
- Full-body dense contact methods retrained in hand versions still underperform HACO by a large margin.
Limitations
- Training memory is high (>26GB) with the ViT-H backbone.
- Ground-truth contact labels depend on dataset-specific distance thresholds chosen by manual inspection.
Experimental setup
Datasets: ObMan DexYCB MOW HO3D H2O3D ARCTIC HOI4D H2O InterHand2.6M HIC PROX RICH
Benchmarks: MOW HIC RICH Hi4D DexYCB
Metrics: precision recall F1-score Intersection volume MPJPE MPVPE Chamfer distance F-score
Baselines: POSA BSTRO DECO DeepContact EasyHOI Focal loss LDAM loss Asymmetric loss
Builds on: class-balanced loss HaMeR ViTPose DECO
Closest papers by meaning
- Understanding Human Hands in Contact at Internet Scale (CVPR 2020 · similarity 0.90)
- How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions (CVPR 2025 · similarity 0.87)
- Learning Explicit Contact for Implicit Reconstruction of Hand-Held Objects from Monocular Images (AAAI 2024 · similarity 0.87)
- VolCo: Volumetric Contact for High-Fidelity Human Grasp Generation (NeurIPS 2026 · similarity 0.87)
- SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-training (ICLR 2025 · similarity 0.86)
- BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.