Research map / 3D vision / Gaussian splatting
BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BIGS, a 3D Gaussian splatting pipeline that reconstructs two hands and an unknown object from a monocular video, using MANO-based shared hand Gaussians, SDS-based diffusion priors for object completion, and a contact-aware joint optimization.
Problem
Category-agnostic 3D reconstruction of bimanual hand-object interactions from monocular RGB video, where severe occlusions between two hands and an unknown object make reconstruction hard; prior methods handle only one hand or pre-known object templates.
Approach
Builds separate hand and object 3D Gaussians in canonical space, shares a single right-hand Gaussian (flipped) across both hands, uses MANO priors for hands and SDS loss with a personalized text-to-image diffusion model (with PiDi ControlNet) to complete unseen object parts, then jointly refines hand translations with a contact regularizer.
Key results
- On ARCTIC, two-hand setting, contact CDr drops from 117.72 (HOLD) to 31.28 cm^2 and object F10 rises from 63.92% to 83.93%.
- On HO3Dv3 limited view, CDr falls from 95.63 (HOLD) to 21.70 cm^2 and F10 rises from 74.90% to 84.02%.
- Rendering on ARCTIC two-hand setting improves PSNR from 12.83 (HOLD) to 24.87 and SSIM from 0.66 to 0.96.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SDS loss for object Gaussians (ablation) | BIGS · ARCTIC | CDo [cm2]: 1.36 -0.08 | Helped |
| Add contact regularization (ablation) | BIGS · ARCTIC | CDl [cm2]: 96.55 -13.41 | Helped |
| Add temporal smoothness loss Lsmt (ablation) | BIGS · ARCTIC | CDl [cm2]: 83.14 -> 50.78 | Helped |
| Share canonical 3D Gaussians between both hands (ablation) | BIGS · ARCTIC | CDr [cm2]: 35.57 -> 31.28 | Helped |
| Use BIGS instead of HOLD | Hand-object reconstruction on ARCTIC · ARCTIC | CDo [cm2]: 2.07 -> 1.28 | Helped |
| Use BIGS instead of HOLD | Hand-object reconstruction on HO3Dv3 Limited View · HO3Dv3 Limited View | CDr [cm2]: 95.63 -> 21.70 | Helped |
| Use BIGS instead of HOLD | Rendering · Two-Hand Settings | PSNR: 12.83 -> 24.87 | Helped |
Limitations
- Limited to articulated hands with known LBS and rigid objects; articulated unknown objects are not handled.
- Object output is a point cloud, so meshes require convex hull approximation for visualization.
- Evaluated only on ARCTIC and HO3Dv3 subsets with limited sequences.
Experimental setup
Datasets: ARCTIC HO3Dv3
Benchmarks: ARCTIC HO3Dv3
Metrics: MPJPE Chamfer distance F10 hand-relative Chamfer distance PSNR SSIM LPIPS
Baselines: HOLD HaMeR iHOI DiffHOI HOMan HLoc
Builds on: 3D Gaussian splatting score distillation sampling MANO HOLD ControlNet
Closest papers by meaning
- HOGSA: Bimanual Hand-Object Interaction Understanding with 3D Gaussian Splatting Based Data Augmentation (AAAI 2025 · similarity 0.92)
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.92)
- Learning Interaction-aware 3D Gaussian Splatting for One-shot Hand Avatars (NeurIPS 2024 · similarity 0.90)
- Single-view Image to Novel-view Generation for Hand-Object Interactions (AAAI 2025 · similarity 0.89)
- EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild (CVPR 2025 · similarity 0.89)
- MANUS: Markerless Grasp Capture using Articulated 3D Gaussians (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.