Research map / 3D vision / 3D generation
InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces InterAct, a consolidated 21.81-hour 3D whole-body human-object interaction dataset with GPT-4-assisted text annotations, optimization-based correction and contact-invariant augmentation (InterAct-X, 30.70 h), and a unified multi-task diffusion benchmark over six HOI tasks.
Problem
Existing 3D human-object interaction datasets are small, inconsistently represented, poorly annotated, and contain artifacts such as penetration, floating contacts, and incorrect hand poses, which limits HOI generation.
Approach
Consolidate seven public MoCap datasets into a unified marker-based representation, correct artifacts with a gradient-based optimization framework, augment data via a contact-invariance principle (object displacement plus human realignment), and train a single multi-task diffusion model that jointly predicts motion and human-object contact.
Key results
- Multi-task marker-based model reduces FID from 3.566 (HOI-Diff) to 2.161 on action-conditioned generation on the InterAct test set.
- Object-conditioned human generation F1 improves from 0.80 (OMOMO 2-stage) to 0.85 with the contact-regression single-stage model (Ours Cont.).
- Training PhysHOI on corrected data raises imitation success rate from 84.4% to 90.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply interaction correction | BEHAVE data · BEHAVE | Contact Ratio: 0.048 -> 0.071 | Helped |
| Apply interaction correction | OMOMO data · OMOMO | Penetration (m): 0.009 -> 0.007 | Helped |
| Use corrected rather than raw interaction data for imitation training | PhysHOI · Interaction imitation | Success rate: 84.4% -> 90.7% | Helped |
| Add contact generation (ablation) | Text-conditioned interaction generation · Text-conditioned interaction generation | FID: 3.192 -> 1.570 | Helped |
| Use BPS object encoding (ablation) | Text-conditioned interaction generation · Text-conditioned interaction generation | FID: 1.997 -> 1.567 | Helped |
| Use an HOI-aware text encoder (ablation) | Text-conditioned interaction generation · Text-conditioned interaction generation | FID: 1.837 -> 1.567 | Helped |
| Add contact guidance (ablation) | Text-conditioned interaction generation · Text-conditioned interaction generation | FID: 1.567 -> 1.567 | No clear effect |
| Use marker-based human representation instead of SMPL (ablation) | Text-to-interaction generation · Text-to-interaction (BEHAVE) | Penetration (m): 0.030 -> 0.025 | Helped |
| Use continuous rather than discrete contact representation (ablation) | Object-conditioned human generation · Object-conditioned human generation | MPMPE: 36.95 -> 35.69 | Helped |
| Add multi-task contact modeling (ablation) | Human-conditioned object generation · Human-conditioned object generation | Terr: 25.92 -> 23.98 | Helped |
| Train the interaction prediction model on InterAct instead of BEHAVE | InterDiff, model size ≃3 · Interaction prediction | Global MPMPE: 0.113 -> 0.091 | Helped |
Surprising findings
- Joint-based representation performs worse than SMPL and marker representations on penetration, showing joints are not ideal for contact modeling.
Limitations
- Only single-human interactions with single dynamic rigid or articulated objects are covered.
- Baseline comparisons limited because few public HOI generation methods were compatible with the setup (HOI-Diff and OMOMO used as base models).
Experimental setup
Datasets: GRAB BEHAVE InterCap Chairs HODome OMOMO IMHD
Benchmarks: InterAct test set BEHAVE subset OMOMO subset
Metrics: R-Precision FID MM Dist Multimodality Diversity MPMPE Foot Sliding contact precision/recall/accuracy/F1 Penetration Contact Ratio
Baselines: HOI-Diff OMOMO (2-stage) InterDiff PhysHOI CHOIS
Builds on: HOI-Diff OMOMO InterDiff PhysHOI HumanML3D
Closest papers by meaning
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.93)
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.92)
- HOI-M^3: Capture Multiple Humans and Objects Interaction within Contextual Environment (CVPR 2024 · similarity 0.92)
- SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios (NeurIPS 2025 · similarity 0.92)
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.91)
- HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusion (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.