Research map / 3D vision / 3D generation
NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NIFTY, which trains an object-conditioned human motion diffusion model guided at sampling time by a learned neural object interaction field, using a synthetic data pipeline that reverses a pre-trained HuMoR motion model from BEHAVE anchor poses.
Problem
Generating realistic 3D human motions that make last-mile contact with objects (sitting, lifting), where paired human-object motion data is scarce and scene-aware models produce penetrations and poor contacts.
Approach
A per-object neural interaction field predicts an offset from a human pose to the valid interaction-pose manifold; its squared norm is used as a differentiable guidance loss at every denoising step of an object-conditioned motion diffusion model. Training data is produced by tree-based rollouts of a time-reversed HuMoR model seeded from a few annotated anchor poses.
Key results
- NIFTY is preferred over SAMP 87.2%, cVAE 89.4%, and cMDM 86.3% of the time in user studies averaged over sitting and lifting.
- For sitting, NIFTY reaches 99.6% D2O within 2cm with 0.00 foot skating and 65.0% penetration-free sequences, versus 31.0% for SAMP.
- Synthetic sitting data scored 4.39 on a 1-5 realism Likert scale versus 4.87 for AMASS mocap.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add learned interaction-field guidance to the object-conditioned diffusion model | cMDM · Sitting | D2O ≤2cm (%): 37.9 -> 99.6 | Helped |
| add learned interaction-field guidance to the object-conditioned diffusion model | cMDM · Lifting | D2O ≤2cm (%): 36.3 -> 77.7 | Helped |
| use an offset-vector interaction field rather than a scalar-distance field (ablation) | NIFTY · Sitting | D2O ≤2cm (%): 80.9 -> 99.6 | Helped |
| use an offset-vector interaction field rather than a scalar-distance field (ablation) | NIFTY · Lifting | D2O ≤2cm (%): 71.1 -> 77.7 | Helped |
| use a learned interaction field rather than a nearest-neighbor field (ablation) | NIFTY · Sitting | D2O ≤2cm (%): 99.8 -> 99.6 | No clear effect |
| use a learned interaction field rather than a nearest-neighbor field (ablation) | NIFTY · Lifting | D2O ≤2cm (%): 77.3 -> 77.7 | Helped |
| use NIFTY instead of SAMP | human-object motion synthesis · Sitting | D2O ≤2cm (%): 80.3 -> 99.6 | Helped |
| use NIFTY instead of cVAE | human-object motion synthesis · Lifting | skeleton distance: 1.70 -> 0.42 | Helped |
Surprising findings
- Synthetic sitting data is preferred by users over NIFTY only about 47.2% of the time, suggesting NIFTY motions are nearly indistinguishable from training data.
- Learning an offset vector field outperforms a scalar distance field on most metrics, which is counter to the intuition that a single distance is a simpler target.
Limitations
- Only sitting and lifting interactions are evaluated; manipulation and chaining of interactions are not addressed.
- Output quality depends on the pre-trained HuMoR motion model; the model sometimes shows backward walking and struggles with harder approaches such as from behind the chair.
- Guided inference is slow (34 s per sample with 10 reps).
Experimental setup
Datasets: BEHAVE AMASS
Benchmarks: BEHAVE-derived held-out test scenes (500 per action)
Metrics: user preference foot skating % D2O within 2cm 95th percentile D2O % penetration <=2cm skeleton distance contact IoU Likert realism score
Baselines: SAMP cVAE (HUMANISE) cMDM
Builds on: Motion Diffusion Model (MDM) HuMoR NSM SAMP Pose-NDF
Closest papers by meaning
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.89)
- DiffGrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model (AAAI 2025 · similarity 0.88)
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.88)
- InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint (NeurIPS 2024 · similarity 0.87)
- Move as You Say Interact as You Can: Language-guided Human Motion Generation with Scene Affordance (CVPR 2024 · similarity 0.87)
- HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusion (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.