Research map / Video / Video generation
Learning to Generate Human-Human-Object Interactions from Textual Descriptions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Human-Human-Object Interaction (HHOI) generation from text by combining separately trained score-based HOI and HHI diffusion models with guided sampling (inconsistency and collision losses), trained on a newly captured multi-view dataset plus image-diffusion-synthesized data.
Problem
Generating realistic interactions among two or more people jointly interacting with a shared object from text, where existing HOI datasets are single-human and HHI datasets lack object context.
Approach
Decompose HHOI into HOI and HHI score-based diffusion models trained separately, then couple them at sampling time with an inconsistency loss (consistency of shared human states) and a capsule-based collision loss, extending to arbitrary numbers of humans.
Key results
- Dyadic HHOI: body pose FD 0.1755 vs 0.6834 (Depth Opt.) and 1.3500 (GenZI); distance FD 0.0689 vs 0.4180 and 0.3542.
- Multi-human (5 humans) success rate 100% vs 22% for GenZI and 13% for Depth Opt.
- User study preference 60.9% for the proposed method vs 20.8% and 18.4% for baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed guided HHOI generation method instead of the strongest baseline | Dyadic HHOI generation · Dyadic HHOI | Body Pose FD: 0.6834 -> 0.1755 | Helped |
| Use the proposed guided HHOI generation method instead of the strongest baseline | Dyadic HHOI generation · Dyadic HHOI | Distance FD: 0.3542 -> 0.0689 | Helped |
| Use the proposed guided HHOI generation method instead of the strongest baseline | Dyadic HHOI generation · Dyadic HHOI | CLIP Score: 0.2647 -> 0.2695 | Helped |
| Use the proposed guided HHOI generation method instead of the strongest baseline | Dyadic HHOI generation · Dyadic HHOI | User Study (%): 20.8 -> 60.9 | Helped |
| Use the proposed method instead of the strongest baseline | Multi-human HHOI generation · 3-human HHOI | CLIP Score: 0.2584 -> 0.2685 | Helped |
| Use the proposed method instead of the strongest baseline | Multi-human HHOI generation · 3-human HHOI | Success Rate (%): 67.3 -> 100.0 | Helped |
| Use the proposed method instead of the strongest baseline | Multi-human HHOI generation · 4-human HHOI | Success Rate (%): 41.4 -> 100.0 | Helped |
| Use the proposed method instead of the strongest baseline | Multi-human HHOI generation · 5-human HHOI | Success Rate (%): 22 -> 100.0 | Helped |
| Use the proposed method instead of the strongest baseline | HHOI generation · 2-human HHOI | Human-Human Penetration Ratio (×1000): 13.91 -> 7.90 | Helped |
| Use the proposed method instead of the strongest baseline | HHOI generation · 3-human HHOI | Human-Human Penetration Ratio (×1000): 14.06 -> 15.68 | Mixed |
| Use the proposed method instead of the strongest baseline | HHOI generation · 2-human HHOI | Human-Object Penetration Ratio (×1000): 11.48 -> 5.49 | Helped |
| Use the proposed method instead of the strongest baseline | HHOI generation · 3-human HHOI | Human-Object Penetration Ratio (×1000): 4.15 -> 7.74 | Hurt |
Surprising findings
- CLIP scores are close across all methods (0.2633-0.2695), so the authors argue FD metrics are more reliable for realism than CLIP score.
- Depth Opt. achieves lower penetration ratios in some multi-human cases, but the authors attribute this to humans being placed unrealistically far from the object.
Limitations
- The score-based model cannot directly learn from HOI-only or HHI-only datasets due to distributional discrepancies in human configurations.
- Background scene rendering makes CLIP scores similar across methods, reducing its discriminative value.
- Dataset is small (about 13.7K HOI and 13.7K HHI samples across 19 categories) and the new capture data is not stated as released.
Experimental setup
Datasets: CORE4D multi-view HHOI capture dataset synthetic HOI/HHI from Flux and ComA 922K human body pose data
Benchmarks: CORE4D test set multi-view HHOI capture test set
Metrics: Fréchet Distance (body pose) Fréchet Distance (interpersonal distance) CLIP score success rate penetration ratio contact distance user study preference
Baselines: GenZI Depth Opt.
Builds on: score-based generative modeling (SDE) GenPose GenZI Diffusion-Noise-Optimization ComA
Closest papers by meaning
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.90)
- HOI-M^3: Capture Multiple Humans and Objects Interaction within Contextual Environment (CVPR 2024 · similarity 0.90)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.90)
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.90)
- ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generation (CVPR 2025 · similarity 0.89)
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.