Research map / 3D vision / 3D generation
HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HOI-Dyn, a driver-responder HOI generation framework that trains a lightweight transformer interaction dynamics model as an auxiliary, training-only residual loss on top of CHOIS-style conditional motion diffusion.
Problem
Generating realistic 3D human-object interactions where object motion is physically plausible and causally consistent with human actions, since existing methods treat human and object motion independently.
Approach
Frame HOI generation as a driver-responder system: a learned interaction dynamics model predicts object rigid-body response from human relative motion, and a residual-based dynamics loss (difference of dynamics residuals on generated vs ground-truth motion) supervises the diffusion model, cancelling model bias. The dynamics model is used only during training.
Key results
- On FullBodyManipulation, HOI-Dyn reports FID 0.48 vs CHOIS 0.87 and FS 0.37 vs 0.41, with C_F1 0.71 vs 0.66.
- On 3D-FUTURE (unseen objects), HOI-Dyn improves FS to 0.37 (CHOIS 0.42) and FID to 1.62 (CHOIS 1.67), with C% 0.54 vs 0.47.
- The dynamics model has 0.5M parameters and about 0.2 GFLOPs, and is used only during training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add interaction dynamics modeling | HOI-Dyn vs CHOIS · FullBodyManipulation test | FS: 0.41 -> 0.37 | Helped |
| add interaction dynamics modeling | HOI-Dyn vs CHOIS · FullBodyManipulation test | C_F1: 0.66 -> 0.71 | Helped |
| add interaction dynamics modeling | HOI-Dyn vs CHOIS · FullBodyManipulation test | T_xy: 3.03 -> 3.26 | Hurt |
| add interaction dynamics modeling | HOI-Dyn vs CHOIS · 3D-FUTURE | C%: 0.47 -> 0.54 | Helped |
| add interaction dynamics modeling | HOI-Dyn vs CHOIS · 3D-FUTURE | P_hand: 0.19 -> 0.26 | Hurt |
| add feet-floor guidance (ablation) | HOI-Dyn without guidance · FullBodyManipulation test | FID: 3.49 -> 0.66 | Helped |
| add hand-object guidance (ablation) | HOI-Dyn without guidance · FullBodyManipulation test | C_F1: 0.60 -> 0.71 | Helped |
| add hand-object guidance (ablation) | HOI-Dyn without guidance · FullBodyManipulation test | FID: 3.49 -> 2.55 | Hurt |
| remove object waypoints (ablation) | HOI-Dyn · FullBodyManipulation test, no waypoints | FS: 0.401 -> 0.376 | Helped |
| remove object waypoints (ablation) | HOI-Dyn · FullBodyManipulation test, no waypoints | Penetration: 0.581 -> 0.582 | Hurt |
| replace residual dynamics loss with non-residual dynamics loss (ablation) | HOI-Dyn · FullBodyManipulation test, no guidance | P_hand: 0.61 -> 0.71 | Hurt |
| use residual dynamics loss instead of no interaction dynamics loss (ablation) | HOI-Dyn · FullBodyManipulation test, no guidance | C_F1: 0.54 -> 0.60 | Helped |
Surprising findings
- Hand-object penetration is slightly worse than CHOIS on 3D-FUTURE (0.26 vs 0.19), and on FullBodyManipulation the hand penetration score is comparable rather than best.
Limitations
- Assumes rigid objects and uses the SMPL-X model, which gives coarse hand representations, causing minor hand-object rotation inaccuracies.
- Dynamics model may not capture object rotations with high precision; the residual-based loss theory relies on Lipschitz and local-smoothness assumptions that are only empirically checked.
Experimental setup
Datasets: FullBodyManipulation 3D-FUTURE
Benchmarks: FullBodyManipulation 3D-FUTURE
Metrics: T_s T_e T_xy H_feet foot sliding FID C_F1 C% P_hand MPJPE
Baselines: CHOIS InterDiff MDM OMOMO (Pred-OMOMO) Lin-OMOMO GT-OMOMO
Builds on: CHOIS OMOMO MDM
Closest papers by meaning
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.91)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.91)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.91)
- Auto-Regressive Diffusion for Generating 3D Human-Object Interactions (AAAI 2025 · similarity 0.89)
- Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos (NeurIPS 2026 · similarity 0.89)
- Learning to Generate Human-Human-Object Interactions from Textual Descriptions (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.