Research map / 3D vision / Gaussian splatting
HOGSA: Bimanual Hand-Object Interaction Understanding with 3D Gaussian Splatting Based Data Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HOGSA, a mesh-based 3D Gaussian Splatting data augmentation framework with a pose optimization module and super-resolution module to expand bimanual hand-object interaction datasets (Arctic, H2O) and improve baseline pose/contact estimation.
Problem
Bimanual hand-object interaction understanding is limited by scarce, occlusion-heavy, low-diversity annotated datasets, and existing augmentation methods lack realism, pose diversity, or efficiency.
Approach
Combine mesh-based 3DGS models of both hands and the object with a bimanual extension of GraspTTA-style pose optimization to generate diverse feasible interaction poses, and a StyleUNet-based super-resolution module (GAN+VGG+L1) to sharpen rendered images before fine-tuning baselines.
Key results
- On Arctic, ArcticNet-SF fine-tuned with HOGSA reduces CDev from 41.35 (Arctic) to 35.23 and raises SR from 71.77 to 77.85.
- On H2O, adding HOGSA lowers CDev from 35.74 to 32.27 and MPJPE from 34.38 to 32.24.
- SRM raises PSNR on Arctic from 35.07 to 36.53 and reduces LPIPS from 0.0194 to 0.0163; generation takes 0.06s per image versus 4.5s for HOIDiffusion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add HOGSA augmented data | ArcticNet-SF · Arctic | CDev_ho: 41.35 -> 35.23 | Helped |
| Add HOGSA augmented data | ArcticNet-SF · Arctic | MPJPE_h: 23.01 -> 20.96 | Helped |
| Add HOGSA augmented data | ArcticNet-SF · Arctic | SR: 71.77 -> 77.85 | Helped |
| Add HOGSA augmented data | ArcticNet-SF · H2O | CDev_ho: 35.74 -> 32.27 | Helped |
| Add HOGSA augmented data | ArcticNet-SF · H2O | MPJPE_h: 34.38 -> 32.24 | Helped |
| Add HOGSA augmented data | InterField-SF · Arctic | Average Distance Error: 9.63/9.91 -> 9.30/8.98 | Helped |
| Add HOGSA augmented data | InterField-SF · H2O | Average Distance Error: 7.75/10.86 -> 7.54/9.87 | Helped |
| Add HOGSA augmented data | ArcticNet-SF · Arctic | CDev_ho: 39.19 -> 35.23 | Helped |
| Add SRM to HOGS | HOGS · Arctic | PSNR: 35.07 -> 36.53 | Helped |
| Add SRM to HOGS | HOGS · Arctic | SSIM: 0.988 -> 0.988 | No clear effect |
| Add SRM to HOGS | HOGS · Arctic | LPIPS: 0.0194 -> 0.0163 | Helped |
| Add SRM to HOGS | HOGS · H2O | SSIM: 0.987 -> 0.986 | Hurt |
Surprising findings
- The HOIDiffusion-based augmentation (Arctic+HOID) only slightly improves the baseline, and on Arctic's ACC metric it is marginally better than HOGSA's left-hand value.
Limitations
- Pose optimization for the articulated Arctic objects uses a restricted angle range, and the bimanual extension is built on single-hand GraspTTA components.
- Augmentation is applied only to two benchmarks (Arctic and H2O) and two baselines, so generality is untested.
- Reported improvements are modest on several metrics (e.g., H2O MDev 5.31 to 5.16).
Experimental setup
Datasets: Arctic H2O COCO2017
Benchmarks: Arctic H2O
Metrics: CDev MRRPE MDev ACC MPJPE AAE SR average distance error PSNR SSIM
Baselines: ArcticNet-SF InterField-SF HOIDiffusion point-cloud 3DGS
Builds on: 3D Gaussian Splatting GaMeS GraspTTA StyleAvatar Arctic
Closest papers by meaning
- BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting (CVPR 2025 · similarity 0.92)
- Learning Interaction-aware 3D Gaussian Splatting for One-shot Hand Avatars (NeurIPS 2024 · similarity 0.90)
- Single-view Image to Novel-view Generation for Hand-Object Interactions (AAAI 2025 · similarity 0.88)
- MANUS: Markerless Grasp Capture using Articulated 3D Gaussians (CVPR 2024 · similarity 0.88)
- BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects (CVPR 2025 · similarity 0.88)
- GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.