Research map / 3D vision / 3D generation
Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a semi-automatic pipeline that annotates 2.5k+ in-the-wild 2D HOI images with 3D human/object meshes and contacts (Open3DHOI), and proposes a training-free 3D Gaussian Splatting optimizer (HOI-Gaussian) for single-image 3D HOI reconstruction.
Problem
3D human-object interaction reconstruction from single in-the-wild images is limited by the lack of 3D HOI data with diverse, open-vocabulary objects; prior 3D HOI datasets are indoor or use few CAD objects.
Approach
Reconstruct 3D object assets from existing 2D HOI images using image-to-3D models and human mesh recovery, refine them manually with annotation tools, and optimize human-object poses with a 3D Gaussian Splatting optimizer whose opacity cues identify contact regions.
Key results
- HOI-Gaussian optimizer reduces object translation error from 77.79 cm (PHOSA) to 38.44 cm and rotation error from 0.95 to 0.41 on Open3DHOI.
- Full model lowers Co2 score to 0.181 versus 0.431 for PHOSA, with contact error 0.128 versus 0.326.
- PointLLM reaches Top-1 0.47 on action recognition given object names, and ChatPose yields MPJPE around 103 on the HOI pose task, indicating current 3D models struggle with 3D HOI understanding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace PHOSA silhouette-based optimization with Gaussian-HOI optimization | 3D HOI reconstruction · Open3DHOI test set | Rotation: 0.95 -> 0.41 | Helped |
| add HOI loss (ablation) | Gaussian-HOI optimizer · Open3DHOI test set | Scale: 0.25 -> 0.16 | Helped |
| add HOI loss (ablation) | Gaussian-HOI optimizer · Open3DHOI test set | Translation (cm): 38.66 -> 38.44 | Helped |
| add HOI loss (ablation) | Gaussian-HOI optimizer · Open3DHOI test set | Rotation: 0.45 -> 0.41 | Helped |
| add HOI loss (ablation) | Gaussian-HOI optimizer · Open3DHOI test set | Cf Distance (cm): 16.9 -> 19.3 | Hurt |
| add depth, collision, and contact losses to Gaussian optimization (ablation) | 3D HOI reconstruction · Open3DHOI test set | Co2: 0.287 -> 0.181 | Helped |
| add contact loss (ablation) | Gaussian optimization with depth and collision losses · Open3DHOI test set | Contact: 0.143 -> 0.128 | Helped |
| add contact loss (ablation) | Gaussian optimization with depth and collision losses · Open3DHOI test set | Collision: 0.045 -> 0.053 | Hurt |
| include the object name in the prompt (ablation) | PointLLM · Open3DHOI HOI understanding | Top-1: 0.20 -> 0.47 | Helped |
| include the action name in the prompt (ablation) | PointLLM · Open3DHOI HOI understanding | Top-1: 0.31 -> 0.32 | No clear effect |
| include the action in the prompt (ablation) | ChatPose · Open3DHOI HOI Pose Chat | MPJPE: 105.2 -> 103.6 | Helped |
| include the action in the prompt (ablation) | ChatPose · Open3DHOI HOI Pose Chat | MPVPE: 133.5 -> 131.2 | Helped |
Surprising findings
- Adding the contact loss slightly increased collision in some images, so the paper introduced a combined Co2 metric to balance the trade-off.
- Giving the action name does not improve object answering for PointLLM, indicating limited true interaction understanding.
Limitations
- Annotation relies on manual filtering and refinement, which limits scalability.
- Contact regions are annotated at 34-part level rather than vertex level.
- Reconstruction quality depends on the accuracy of image-to-3D and human pose estimation tools.
Experimental setup
Datasets: HAKE-Large SWIG-HOI
Benchmarks: Open3DHOI
Metrics: scale error translation error (cm) rotation error Chamfer distance Co2 score collision contact Top-1 accuracy MPJPE MPVPE
Baselines: PHOSA Coarse Recon
Builds on: 3D Gaussian Splatting GauHuman PHOSA InstantMesh
Closest papers by meaning
- End-to-End HOI Reconstruction Transformer with Graph-based Encoding (CVPR 2025 · similarity 0.90)
- PICO: Reconstructing 3D People In Contact with Objects (CVPR 2025 · similarity 0.90)
- HOI-M^3: Capture Multiple Humans and Objects Interaction within Contextual Environment (CVPR 2024 · similarity 0.90)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.89)
- Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos (NeurIPS 2026 · similarity 0.89)
- Open-World Human-Object Interaction Detection via Multi-modal Prompts (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.