Research map / 3D vision / 3D generation
BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BOTH57M, a multi-modal body-and-hand motion dataset with finger-level text annotations, and BOTH2Hands, a two-stage diffusion plus cross-attention model that generates two-hand motions from body dynamics and text prompts.
Problem
Two-hand motion generation is data-scarce and existing methods handle only body-to-hand or text-to-body conditions, not hybrid body-dynamics plus text prompts for hands.
Approach
Train separate body-to-hand and text-to-hand diffusion denoisers, project text-conditioned local hands into the body's global coordinate frame via forward kinematics, then blend the two hand streams with an alternating cross-attention transformer trained with a weighted blending loss.
Key results
- BOTH2Hands reaches R-Precision Top1 0.037 and FID 0.201 on the BOTH57M test split, versus 0.034 and 0.181 for real data and 0.042/0.461 for T2M-GPT's Top1/FID.
- Training on BOTH57M and testing on Motion-X gives FID 0.858 versus 2.399 for a model trained on Motion-X and tested on BOTH57M, indicating the dataset transfers well.
- Removing the blending loss raises FID from 0.201 to 0.210 and removing hand projection raises it to 0.204 in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add hand projection (ablation) | BOTH2Hands · BOTH57M test | FID: 0.204±.026 -> 0.201±.020 | Helped |
| Add hand projection (ablation) | BOTH2Hands · BOTH57M test | R Precision: 0.036±.014 -> 0.037±.014 | Helped |
| Add blending loss instead of linear distance loss (ablation) | BOTH2Hands · BOTH57M test | FID: 0.210±.022 -> 0.201±.020 | Helped |
| Add blending loss instead of linear distance loss (ablation) | BOTH2Hands · BOTH57M test | R Precision: 0.034±.020 -> 0.037±.014 | Helped |
| Train on BOTH57M rather than Motion-X | BOTH2Hands · BOTH57M test | FID: 2.399±.076 -> 0.201±.020 | Helped |
| Train on BOTH57M rather than Motion-X | BOTH2Hands · BOTH57M test | MModality: 0.581±.090 -> 1.312±.034 | Helped |
Surprising findings
- Standard metrics such as R-Precision show only marginal gains for the multi-conditioned model, which the authors attribute to hands being a small portion of the full body and non-linear hand-text correlation.
Limitations
- Standard motion metrics (R-Precision, FID) are insensitive to multi-conditioned hand quality, so gains are marginal.
- Evaluation relies on a random split of a single dataset of 1,384 clips with 32-camera capture, limiting scale and diversity.
- Hand motion constitutes a small portion of full-body metrics, which limits measured improvements.
Experimental setup
Datasets: BOTH57M Motion-X
Benchmarks: BOTH57M test split Motion-X test split
Metrics: R Precision FID MM-Dist Diversity MModality
Baselines: T2M-GPT MDM MLD EgoEgo
Builds on: DDPM MDM EgoEgo InterGen SMPL-H
Closest papers by meaning
- Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction (CVPR 2024 · similarity 0.90)
- HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Models (AAAI 2025 · similarity 0.90)
- HumanTOMATO: Text-aligned Whole-body Motion Generation (ICML 2024 · similarity 0.90)
- How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions (CVPR 2025 · similarity 0.88)
- GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities (CVPR 2025 · similarity 0.87)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.