Research map / 3D vision / Gaussian splatting
MANUS: Markerless Grasp Capture using Articulated 3D Gaussians
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MANUS, a markerless hand-object grasp capture method built on an articulated 3D Gaussian splatting hand model (MANUS-Hand) and a Gaussian object model, plus the MANUS-Grasps multi-view dataset with wet-paint ground-truth contacts.
Problem
Accurately capturing hand-object grasps and estimating hand-object contact from markerless multi-view RGB, since skeleton, mesh, and parametric hand models do not represent hand shape accurately enough for contact estimation.
Approach
Model the articulated hand and the object as 3D Gaussians with explicit positions and orientations, skin them with forward kinematics and grid-based linear blend skinning, and estimate contact from proximity between hand and object Gaussians; validate contacts against wet-paint transfer ground truth.
Key results
- On 15 wet-paint evaluation sequences, MANUS achieves mIoU 0.206/0.152/0.275 and F1 0.335/0.251/0.424 for Subjects 1-3, beating MANO and HARP on every subject.
- On InterHand2.6M, MANUS-Hand gets PSNR 26.32 and SSIM 0.9872, above TAVA (22.85, 0.983) but below LiveHand (31.16 PSNR), with much lower test time than TAVA (0.049s vs 11.00s).
- Contact accuracy degrades as camera views drop from 30+ to 5 (e.g., Subject 1 mIoU 0.206 to 0.147), supporting the need for dense views.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MANUS instead of HARP for grasp capture | MANUS · MANUS-Grasps, three subjects | mIoU: Subject1: 0.173 -> 0.206; Subject2: 0.148 -> 0.152; Subject3: 0.224 -> 0.275 | Helped |
| Use MANUS instead of HARP for grasp capture | MANUS · MANUS-Grasps, three subjects | F1 score: Subject1: 0.28875 -> 0.335; Subject2: 0.2474 -> 0.251; Subject3: 0.361 -> 0.424 | Helped |
| Use MANUS-Hand instead of LiveHand | MANUS-Hand · InterHand2.6M | PSNR: 31.16 -> 26.32 | Hurt |
| Use MANUS-Hand instead of LiveHand | MANUS-Hand · InterHand2.6M | SSIM: 0.9818 -> 0.9872 | Helped |
| Use MANUS-Hand instead of LiveHand | MANUS-Hand · InterHand2.6M | LPIPS: 0.0278 -> 0.068 | Hurt |
| Use MANUS-Hand instead of LiveHand | MANUS-Hand · InterHand2.6M | Test time (s): 0.022 -> 0.049 | Hurt |
| Use 5 instead of 30+ camera views (ablation) | MANUS · MANUS-Grasps, Subjects 1–3 | mIoU: Subject1: 0.147 vs 0.206; Subject2: 0.140 vs 0.152; Subject3: 0.214 vs 0.275 | Hurt |
| Use 10 instead of 30+ camera views (ablation) | MANUS · MANUS-Grasps, Subjects 1–3 | mIoU: Subject1: 0.164 vs 0.206; Subject2: 0.145 vs 0.152; Subject3: 0.256 vs 0.275 | Hurt |
| Use 20 instead of 30+ camera views (ablation) | MANUS · MANUS-Grasps, Subjects 1–3 | mIoU: Subject1: 0.176 vs 0.206; Subject2: 0.142 vs 0.152; Subject3: 0.261 vs 0.275 | Hurt |
| Use fewer camera views instead of 30+ (ablation) | MANUS · MANUS-Grasps, Subjects 1–3 | F1 score: 5 views: 0.244, 0.235, 0.343; 10 views: 0.266, 0.242, 0.401; 20 views: 0.271, 0.240, 0.410; 30+ views: 0.335, 0.251, 0.4 | Hurt |
Surprising findings
- LiveHand achieves higher PSNR (31.16) and lower LPIPS than MANUS-Hand on InterHand2.6M, so MANUS-Hand is not the best visual-quality model; the authors argue visual quality is not its goal.
Limitations
- Focuses on single-hand grasping with static objects; no modeling of skin-stretch deformation.
- Does not address hand-object manipulation over longer time frames.
- Requires dense multi-view capture (tens of cameras) for accurate results; contact accuracy drops with fewer views.
Experimental setup
Datasets: MANUS-Grasps InterHand2.6M
Benchmarks: MANUS-Grasps wet-paint evaluation sequences InterHand2.6M
Metrics: mIoU F1 score PSNR SSIM LPIPS test time
Baselines: MANO HARP LiveHand TAVA
Builds on: 3D Gaussian splatting Fast-SNARF MANO AlphaPose Segment Anything Model
Closest papers by meaning
- BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting (CVPR 2025 · similarity 0.88)
- HOGSA: Bimanual Hand-Object Interaction Understanding with 3D Gaussian Splatting Based Data Augmentation (AAAI 2025 · similarity 0.88)
- HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction (NeurIPS 2025 · similarity 0.87)
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.87)
- LatentHOI: On the Generalizable Hand Object Motion Generation with Latent Hand Diffusion. (CVPR 2025 · similarity 0.87)
- Single-view Image to Novel-view Generation for Hand-Object Interactions (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.