Research map / 3D vision / 3D generation

Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos

NeurIPS 2026 · 3D vision › 3D generation · approach: Hand-object interaction generation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HA-HOI, a training-free human-first, object-follow pipeline that reconstructs 4D human-object interaction from monocular video, refines contact with VLM-based proposals, and projects the result into a physics-based humanoid-object rollout via a residual PPO controller.

Problem

Monocular 4D human-object interaction reconstruction yields visually plausible trajectories that lack stable contact, functional manipulation, and physical plausibility, so they are unsuitable as teacher trajectories for humanoid-object simulation.

Approach

Anchor the reconstruction on the human motion (SMPL-X with MANO hands and calibrated depth), then recover the object relative to the human, refine contact using InteractVLM proposals and staged SDF-based optimization, and finally track with a sequence-specific residual PPO controller in IsaacGym.

Key results

• On BEHAVE, HA-HOI reduces CD-h from 7.51 (CARI4D) to 7.11 and Acc-h from 0.54 (VisTracker) to 0.52.
• Penetration drops from 0.084 (CARI4D) to 0.013 cm; removing contact optimization raises penetration to 0.097.
• Object Chamfer distance (15.68) is worse than CARI4D (11.93) and VisTracker (12.58), which the authors attribute to template-free object reconstruction.

Surprising findings

• Removing contact refinement marginally improves some visual alignment scores (e.g., Acc-o 0.49 vs 0.59) while greatly increasing penetration, showing surface metrics can favor physically invalid results.
• Template-based baselines achieve lower object Chamfer distance than the template-free HA-HOI.

Limitations

• Reconstruction remains vision-based, so severe occlusion can yield inaccurate body pose or object geometry.
• Physics rollout is only evaluated in simulation; sim-to-real transfer to humanoid robots is not demonstrated.
• Object Chamfer distance is worse than template-based methods.

Experimental setup

Datasets: BEHAVE

Benchmarks: BEHAVE

Metrics: Chamfer distance (human, object, combined) acceleration error penetration

Baselines: CHORE InterTrack HOI-TG VisTracker CARI4D THO

Builds on: FoundationPose InteractVLM SMPL-X PPO CARI4D

Closest papers by meaning

• InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.90)
• CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.89)
• Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactions (CVPR 2025 · similarity 0.89)
• HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusion (NeurIPS 2025 · similarity 0.89)
• HOI-M^3: Capture Multiple Humans and Objects Interaction within Contextual Environment (CVPR 2024 · similarity 0.88)
• End-to-End HOI Reconstruction Transformer with Graph-based Encoding (CVPR 2025 · similarity 0.88)