Research map / 3D vision / 3D generation
GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GigaHands, a 34-hour multi-view (51 cameras) markerless dataset of bimanual hand activities with 14k text-annotated motion clips and 84k text descriptions, and demonstrates it on text-to-motion synthesis, motion captioning, and dynamic radiance field reconstruction.
Problem
Large-scale, diverse, richly annotated 3D bimanual hand activity data is lacking; existing datasets are small, sparse, marker-based in unnatural studios, or poorly annotated for text.
Approach
An instruct-to-annotate pipeline that elicits diverse atomic hand actions via LLM-generated scripts, filmed in a markerless 51-camera studio, followed by automatic hand/object 3D estimation and LLM-based text augmentation to produce paired motion-text data at scale.
Key results
- Text-to-motion models trained on GigaHands achieve FID 4.70 and Diversity 10.5 versus OakInk2-trained FID 19.6, outperforming baselines on most metrics.
- Motion captioning trained on GigaHands reaches Pairwise-BLEU 0.916 and distinct-1 15.3% versus 8.21 and 2.76 for OakInk2, indicating more diverse captions.
- Dataset scale study shows text-to-motion and reconstruction metrics improve consistently from 10% to 100% of training data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the text-driven motion synthesis model on GigaHands rather than OakInk2 | T2M-GPT motion synthesis · Text-driven motion synthesis | FID: 19.6 -> 4.70 | Helped |
| Train the text-driven motion synthesis model on GigaHands rather than OakInk2 | T2M-GPT motion synthesis · Text-driven motion synthesis | MM Dist.: 7.75 -> 6.68 | Hurt |
| Train the text-driven motion synthesis model on GigaHands rather than OakInk2 | T2M-GPT motion synthesis · Text-driven motion synthesis | R-Precision @1 (%): 17.9 -> 31.2 | Helped |
| Increase GigaHands training data from 10% to 100% | T2M-GPT motion reconstruction and text-to-motion generation · GigaHands test set | FID, MM Dist., and Top-1/Top-3 accuracy | Helped |
| Train the motion captioning model on GigaHands rather than TACO | TM2T motion captioning · Motion captioning | distinct-1 (%): 2.03 -> 15.3 | Helped |
| Train the motion captioning model on GigaHands rather than OakInk2 | TM2T motion captioning · Motion captioning | BLEU@4: 39.9 -> 43.1 | Helped |
| Train the motion captioning model on GigaHands rather than TACO | TM2T motion captioning · Motion captioning | ROUGE: 61.2 -> 57.7 | Hurt |
Surprising findings
- Model trained on GigaHands has slightly lower captioning R-Precision@1 (57.0%) than the TACO triplet classification baseline at the same metric scale, despite richer captions.
Limitations
- Studio setting restricts capture to limited space, making large-environment motions hard to record.
- Automatic tracking of articulated and non-rigid objects remains challenging.
- Robotics and HCI downstream applications are proposed but not explored.
Experimental setup
Datasets: GigaHands TACO OakInk2 Ego-Exo4D OakInk-2 action verbs
Benchmarks: GigaHands test split TACO OakInk2
Metrics: R-Precision MM Dist FID Diversity Multimodality BLEU@4 ROUGE BERTScore Pairwise-BLEU distinct-n
Baselines: TACO OakInk2
Builds on: T2M-GPT TM2T HaMeR MANO 2DGS
Closest papers by meaning
- OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion (CVPR 2024 · similarity 0.88)
- TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding (CVPR 2024 · similarity 0.88)
- BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting (CVPR 2025 · similarity 0.88)
- BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics (CVPR 2024 · similarity 0.87)
- BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects (CVPR 2025 · similarity 0.87)
- HOGSA: Bimanual Hand-Object Interaction Understanding with 3D Gaussian Splatting Based Data Augmentation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.