Research map / Image generation / Autoregressive and latent generation
HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HandDiffuse12.5M, a real multi-view two-hand interaction temporal dataset (250K frames), and HandDiffuse, a diffusion-based controllable interacting-hand motion generator with an interaction loss.
Problem
Generating temporally consistent, strongly interacting two-hand motion is blocked by the lack of large temporal two-hand interaction datasets, and existing controllable motion generators do not model hand-hand interaction well.
Approach
A modular diffusion denoiser pair (single-hand denoiser plus interacting-hands denoiser) with local and global motion representations suited to different controllers, plus an Interaction Loss combining a spring-style contact potential and a bone-length shape loss to reduce penetration artifacts.
Key results
- HandDiffuse reaches FID 0.161 and diversity 11.829 in unconditioned generation, versus 0.526 FID for MDM and 0.251 for InterGen.
- With trajectory control, HandDiffuse achieves FID 0.173 and SDF 1.141, the best among compared methods.
- The dataset has 250K temporal frames (about 12.5M images), with a contacting-hands ratio of 57.3% versus 44.74% for InterHand2.6M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use global rather than local representation (ablation) | HandDiffuse · Unconditioned | FID: 0.431 -> 0.161 | Helped |
| Use global rather than local representation (ablation) | HandDiffuse · In-betweening | FID: 0.567 -> 0.273 | Helped |
| Use local rather than global representation (ablation) | HandDiffuse · Trajectory Control | FID: 0.418 -> 0.173 | Helped |
| Add SHDe alongside IHDe (ablation) | HandDiffuse · Unconditioned | FID: 0.168 -> 0.161 | Helped |
| Add SHDe alongside IHDe (ablation) | HandDiffuse · In-betweening | FID: 0.282 -> 0.273 | Helped |
| Add SHDe alongside IHDe (ablation) | HandDiffuse · Trajectory Control | FID: 0.182 -> 0.173 | Helped |
| Add contact potential loss (ablation) | HandDiffuse · Unconditioned | FID: 0.171 -> 0.161 | Helped |
| Add contact potential loss (ablation) | HandDiffuse · In-betweening | FID: 0.317 -> 0.273 | Helped |
| Add contact potential loss (ablation) | HandDiffuse · Trajectory Control | FID: 0.210 -> 0.173 | Helped |
| Add shape loss (ablation) | HandDiffuse · Unconditioned | FID: 0.390 -> 0.161 | Helped |
| Add shape loss (ablation) | HandDiffuse · In-betweening | FID: 0.512 -> 0.273 | Helped |
| Add shape loss (ablation) | HandDiffuse · Trajectory Control | FID: 0.380 -> 0.173 | Helped |
Surprising findings
- InterGen generates static mean-pose motion without conditioning, giving a low FID despite poor diversity (4.928).
Limitations
- Metrics are FID, diversity and SDF only, with no human evaluation or user study reported.
- Models are trained on a single RTX 3090 for about two days, with motion length fixed at 200 frames.
Experimental setup
Datasets: HandDiffuse12.5M InterHand2.6M Re:InterHand
Benchmarks: HandDiffuse12.5M
Metrics: FID Diversity SDF (penetration)
Baselines: MDM InterGen BUDDI InterHandGen
Builds on: MDM DDIM HumanML3D MANO
Closest papers by meaning
- InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion (CVPR 2024 · similarity 0.90)
- BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics (CVPR 2024 · similarity 0.90)
- Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling (AAAI 2024 · similarity 0.90)
- Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction (CVPR 2024 · similarity 0.89)
- DiffGrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model (AAAI 2025 · similarity 0.89)
- SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.