Research map / 3D vision / 3D generation
HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HOIDiffusion, a geometry-conditioned Stable Diffusion model (skeleton, segmentation, normal maps plus text) fine-tuned on DexYCB with background regularization to generate realistic 3D hand-object interaction images.
Problem
Scarcity of 3D hand-object interaction data; off-the-shelf text-to-image models like Stable Diffusion produce physically implausible hand-object interactions (e.g., wrong finger counts, poor grasps) and offer no control over 3D geometry.
Approach
Two-stage pipeline: GrabNet generates grasp trajectories from object meshes, then a Stable Diffusion model fine-tuned with hand skeleton, hand-object mask and normal-map conditions synthesizes images, with an appearance regularization using LLM-prompted background images and classifier-free guidance to keep text editability and disentangle geometry from appearance.
Key results
- HOIDiffusion achieves FID 55.22 vs 63.71 for finetuned LDM and 87.99 for ControlNet on generated hand-object images (Table 1).
- Mean hand contact recall of 95.49% vs 89.85% for ControlNet and 84.33% for Affordance Diffusion (Table 2).
- Using generated HOI images to augment NOCS training improves DualPoseNet IoU@25 from 84.1 to 90.9 and 5deg2cm from 28.0 to 29.2 (Table 3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HOIDiffusion instead of ControlNet | Hand-object image synthesis · DexYCB synthesis | FID: 87.99 -> 55.22 | Helped |
| Use HOIDiffusion instead of ControlNet | Hand-object image synthesis · DexYCB synthesis | CLIPScore: 0.77 -> 0.78 | Helped |
| Use HOIDiffusion instead of ControlNet | Hand-object image synthesis · Hand contact evaluation | Mean contact recall (%): 89.85 -> 95.49 | Helped |
| Use HOIDiffusion instead of ControlNet | Hand-object image synthesis · Hand contact evaluation | PCK: 0.67 -> 0.85 | Helped |
| Train SPD with HOIDiffusion-generated images | SPD · NOCS | 5°5cm: 19.9 -> 23.2 | Helped |
| Train SPD with HOIDiffusion-generated images | SPD · NOCS | IoU@50: 75.3 -> 71.1 | Hurt |
| Train DualPoseNet with HOIDiffusion-generated images | DualPoseNet · NOCS | IoU@50: 79.7 -> 84.1 | Helped |
| Train DualPoseNet with HOIDiffusion-generated images | DualPoseNet · NOCS | 5°5cm: 34.3 -> 34.4 | Helped |
| Add estimated normal maps (ablation) | HOIDiffusion · DexYCB synthesis (1k images) | FID: 82.40 -> 77.64 | Helped |
| Add hand keypoint projection (ablation) | HOIDiffusion · DexYCB synthesis (1k images) | FID: 81.93 -> 77.64 | Helped |
| Add hand-object segmentation (ablation) | HOIDiffusion · DexYCB synthesis (1k images) | FID: 78.57 -> 77.64 | Helped |
| Add appearance regularization (ablation) | HOIDiffusion · Appearance prompt alignment | CLIPScore: 0.66 -> 0.79 | Helped |
Surprising findings
- Plain ControlNet with the same multi-condition input yields a much worse sFID (248.56) than the finetuned unconditional LDM, despite better CLIPScore.
Limitations
- Relies on DexYCB, which has a uniform clean lab background, requiring background regularization to avoid bias.
- Hand grasp poses depend on a pre-trained GrabNet, so generated grasps are limited to its distribution.
- Video generation suffers from flickering that is only partially mitigated by anchor-frame attention.
Experimental setup
Datasets: DexYCB NOCS
Benchmarks: DexYCB NOCS
Metrics: FID sFID Inception Score CLIPScore Hand Contact Recall PCK IoU 5deg2cm/5deg5cm/10deg2cm/10deg5cm
Baselines: LDM (finetuned Stable Diffusion) DreamBooth ControlNet Affordance Diffusion
Builds on: Stable Diffusion ControlNet DreamBooth GrabNet LLaVA
Closest papers by meaning
- Single-view Image to Novel-view Generation for Hand-Object Interactions (AAAI 2025 · similarity 0.90)
- DiffGrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model (AAAI 2025 · similarity 0.89)
- Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction (CVPR 2024 · similarity 0.89)
- How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions (CVPR 2025 · similarity 0.89)
- SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios (NeurIPS 2025 · similarity 0.89)
- HandDiff: 3D Hand Pose Estimation with Diffusion on Image-Point Cloud (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.