Research map / 3D vision / 3D generation
End-to-End HOI Reconstruction Transformer with Graph-based Encoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HOI-TG, an end-to-end transformer with graph residual blocks that implicitly models human-object interaction for joint single-image human and object mesh reconstruction, achieving SOTA on BEHAVE and InterCap.
Problem
Joint 3D reconstruction of human and object meshes from a single image; explicit interaction/contact constraints conflict with global mesh reconstruction and often cause penetration or slow post-optimization.
Approach
Replace explicit human-object contact constraints with an implicit transformer encoder over joint, human-vertex and object-vertex queries (grid-sampled image features plus initial 3D coordinates), and add GCN-based human and object graph residual blocks inside each transformer block to preserve local topology.
Key results
- On BEHAVE, HOI-TG reduces human CD to 4.59 cm and object CD to 8.00 vs CONTHO's 4.99 and 8.42 (about 8.0% and 5.0% gains).
- On InterCap, HOI-TG improves human and object mesh reconstruction by 8.9% and 8.6% over CONTHO, with contact precision 0.700 and recall 0.473.
- Ablation shows initial-mesh grid-sampled queries outperform static template queries (CDhuman 4.59 vs 4.95).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace explicit interaction constraints with HOI-TG's implicit transformer and graph-based encoding | HOI reconstruction · BEHAVE | CDhuman: 8.0% | Helped |
| Replace explicit interaction constraints with HOI-TG's implicit transformer and graph-based encoding | HOI reconstruction · BEHAVE | CDobject: 5.0% | Helped |
| Replace explicit interaction constraints with HOI-TG's implicit transformer and graph-based encoding | HOI reconstruction · BEHAVE | Contactp: 3.4% | Helped |
| Replace explicit interaction constraints with HOI-TG's implicit transformer and graph-based encoding | HOI reconstruction · BEHAVE | Contactr: 5.8% | Helped |
| Replace explicit interaction constraints with HOI-TG's implicit transformer and graph-based encoding | HOI reconstruction · InterCap | CDhuman: 8.9% | Helped |
| Replace explicit interaction constraints with HOI-TG's implicit transformer and graph-based encoding | HOI reconstruction · InterCap | CDobject: 8.6% | Helped |
| Replace explicit interaction constraints with HOI-TG's implicit transformer and graph-based encoding | HOI reconstruction · InterCap | Contactp: 3.9% | Helped |
| Replace explicit interaction constraints with HOI-TG's implicit transformer and graph-based encoding | HOI reconstruction · InterCap | Contactr: 4.1% | Helped |
| Use initial-vertex grid-sampled features instead of static query features (ablation) | HOI-TG · BEHAVE | CDhuman: 4.95 -> 4.59 | Helped |
| Add the Human Graph Residual Block (ablation) | HOI-TG · BEHAVE | CDhuman: 4.73 -> 4.61 | Helped |
| Add the Object Graph Residual Block on top of the Human Graph Residual Block (ablation) | HOI-TG · BEHAVE | CDobject: 8.11 -> 8.00 | Helped |
| Add joint refinement loss and multi-scale human vertex loss together (ablation) | HOI-TG · BEHAVE | CDhuman: 5.01 -> 4.62 | Helped |
Surprising findings
- Adding only the human or only the object loss/graph component alone slightly hurts object reconstruction in the loss ablation, with gains appearing only when combined.
Limitations
- Struggles with lying poses.
- Struggles with purely symmetric objects.
Experimental setup
Datasets: BEHAVE InterCap
Benchmarks: BEHAVE InterCap
Metrics: Chamfer distance (human) Chamfer distance (object) contact precision contact recall F1
Baselines: METRO Graphormer PHOSA CHORE CONTHO
Builds on: METRO Graphormer CONTHO Hand4Whole
Closest papers by meaning
- Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactions (CVPR 2025 · similarity 0.90)
- Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos (NeurIPS 2026 · similarity 0.88)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.88)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.87)
- Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generation (CVPR 2024 · similarity 0.87)
- Guiding Human-Object Interactions with Rich Geometry and Relations (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.