Research map / 3D vision / 3D generation
Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Procedurally generates ~1.1M synthetic human-object interaction images with 21k+ object shapes (ProciGen) and trains a hierarchical image-conditioned point diffusion model (HDM) for template-free human-object reconstruction from a single RGB image.
Problem
Reconstructing 3D human and object jointly from a single RGB image without object templates is limited by the small number of objects in real interaction datasets, which cannot cover the combinatorial space of human-object interactions.
Approach
Transfer contacts from captured interactions to new same-category object instances using dense correspondences from a per-category point-cloud autoencoder, refine poses by contact-based optimization, and then train a two-stage diffusion model (joint coarse prediction followed by separate human/object diffusion refinement with cross-attention) on the resulting data.
Key results
- On BEHAVE, HDM+ProciGen achieves combined F-score 0.4604 vs. PC2 0.4231 and CHORE 0.3966 (template-based).
- On InterCap, HDM+ProciGen achieves combined F-score 0.5344 vs. PC2 0.5057 and CHORE 0.4687.
- Generalization to unseen InterCap objects (trained on ProciGen): HDM+ProciGen reaches 0.3976 combined vs. 0.2127 for HDM trained on BEHAVE only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ProciGen training data to HDM (ablation) | HDM · BEHAVE test | Combined F-score@0.01m: 0.4214 -> 0.4604 | Helped |
| Add ProciGen training data to HDM (ablation) | HDM · BEHAVE test | Human F-score@0.01m: 0.3605 -> 0.3925 | Helped |
| Add ProciGen training data to HDM (ablation) | HDM · BEHAVE test | Object F-score@0.01m: 0.4575 -> 0.5049 | Helped |
| Train with ProciGen instead of BEHAVE-only data (ablation) | PC2 · BEHAVE test | Combined F-score@0.01m: 0.4231 -> 0.4486 | Helped |
| Add ProciGen training data (ablation) | HDM · InterCap unseen objects | Combined F-score@0.01m: 0.2127 -> 0.3976 | Helped |
| Add random shape augmentation (ablation) | HDM trained on BEHAVE · InterCap unseen objects | Combined F-score@0.01m: 0.2127 -> 0.2680 | Helped |
| Replace HDM's segmentation with projected 2D masks (ablation) | HDM · BEHAVE chair test images | Combined F-score@0.01m: 0.3875 -> 0.3776 | Hurt |
| Replace hierarchical diffusion with a single joint model (ablation) | HDM · BEHAVE chair test images | Combined F-score@0.01m: 0.3875 -> 0.3743 | Hurt |
| Combine separate models using direct image-predicted translation and scale (ablation) | HDM · BEHAVE chair test images | Combined F-score@0.01m: 0.3875 -> 0.3380 | Hurt |
| Add cross attention between human and object refinement models (ablation) | HDM · BEHAVE chair test images | Combined F-score@0.01m: 0.3807 -> 0.3875 | Helped |
| Add cross attention between human and object refinement models (ablation) | HDM · BEHAVE chair test images | Object F-score@0.01m: 0.3806 -> 0.3916 | Helped |
Surprising findings
- Training only on synthetic ProciGen data performs on par with CHORE, which uses object templates, on BEHAVE and InterCap.
Limitations
- Assumes known 2D human and object segmentation masks as input.
- Generalization to in-the-wild images is shown only qualitatively (COCO).
- Procedural pipeline relies on a seed set of captured interactions and per-category aligned shapes (ART used when unaligned).
Experimental setup
Datasets: BEHAVE InterCap ShapeNet Objaverse ABO MGN
Benchmarks: BEHAVE InterCap COCO (qualitative)
Metrics: F-score@0.01m
Baselines: CHORE PC2
Builds on: PC2 PVCNN CHORE SMPL-H PointNet
Closest papers by meaning
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.89)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.89)
- Generative Proxemics: A Prior for 3D Social Interaction from Images (CVPR 2024 · similarity 0.89)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.89)
- Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models (NeurIPS 2024 · similarity 0.88)
- Reconstructing Close Human Interaction with Appearance and Proxemics Reasoning (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.