Research map / 3D vision / Human pose estimation
Joint Reconstruction of 3D Human and Object via Contact-Based Refinement Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CONTHO, which jointly reconstructs 3D human and object from a single image by estimating human-object contact with 3D-guided contact estimation and refining meshes with a contact-masked Transformer (CRFormer).
Problem
Joint 3D reconstruction of humans and objects from a single image, where existing methods either estimate contact separately or rely on pre-defined contact regions and imperfect optimization targets, and learning-based models may learn undesired human-object correlations.
Approach
Uses initially reconstructed 3D human and object meshes as explicit 3D guidance (3D vertex features with grid-sampled image features) for contact estimation, then masks non-contact vertex features before cross/self-attention refinement so that only contact regions drive human-object interaction, suppressing spurious correlations.
Key results
- On BEHAVE, CONTHO reaches contact precision 0.754 and recall 0.587 for human contact estimation, above DECO (0.638/0.337).
- On BEHAVE joint reconstruction, CONTHO achieves CDhuman 4.99 cm and CDobject 8.42 cm, versus CHORE's 5.58 and 10.66.
- On InterCap, CONTHO reduces CDobject to 9.50 cm from CHORE's 12.81 cm and PHOSA's 20.57 cm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use contact-based masking instead of no contact maps (ablation) | CONTHO refinement · BEHAVE | CDhuman: 5.13 -> 4.99 | Helped |
| use contact-based masking instead of adding contact maps (ablation) | CONTHO refinement · BEHAVE | CDobject: 8.54 -> 8.42 | Helped |
| use contact-based masking instead of concatenating contact maps (ablation) | CONTHO refinement · BEHAVE | Contactrec_r: 0.477 -> 0.496 | Helped |
| add CA Transformers to CRFormer (ablation) | CONTHO refinement · BEHAVE | CDhuman: 5.40 -> 4.99 | Helped |
| add SA Transformers to CRFormer (ablation) | CONTHO refinement · BEHAVE | CDobject: 8.88 -> 8.42 | Helped |
Surprising findings
- A naive Transformer baseline that concatenates contact maps with features yields object sensitivity maps focused on human regions unrelated to interaction, showing undesired human-object correlation.
Limitations
- Relies on GT bounding boxes for cropping at training and testing time.
- Relies on DetectronV2 segmentations and a fixed object mesh template for object pose prediction.
Experimental setup
Datasets: BEHAVE InterCap
Benchmarks: BEHAVE InterCap
Metrics: contact precision contact recall Chamfer distance (human) Chamfer distance (object)
Baselines: POSA BSTRO DECO PHOSA CHORE
Builds on: Hand4Whole CHORE BSTRO DECO Transformer (cross/self-attention)
Closest papers by meaning
- PICO: Reconstructing 3D People In Contact with Objects (CVPR 2025 · similarity 0.87)
- Learning Explicit Contact for Implicit Reconstruction of Hand-Held Objects from Monocular Images (AAAI 2024 · similarity 0.87)
- InteractVLM: 3D Interaction Reasoning from 2D Foundational Models (CVPR 2025 · similarity 0.86)
- End-to-End HOI Reconstruction Transformer with Graph-based Encoding (CVPR 2025 · similarity 0.86)
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.86)
- UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.