Research map / Vision-language and multimodal / Robot action and navigation
Grasp2Grasp: Vision-Based Dexterous Grasp Translation via Schrödinger Bridges
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames vision-based dexterous grasp translation across hands as a latent Schrödinger Bridge trained with score and flow matching, using four physics-informed OT ground costs (base pose, contact, wrench, Jacobian).
Problem
Transferring grasp intent from a source robotic hand observed in vision to a target hand with different morphology, without paired demonstrations or hand-specific simulation.
Approach
Model grasp translation as a stochastic transport between source and target latent grasp distributions via a Schrödinger Bridge, trained simulation-free with [SF]^2M on minibatch entropic OT couplings under physics-informed ground costs (pose, contact Chamfer, GWH IoU, Jacobian manipulability) instead of Euclidean/geodesic distances.
Key results
- Jacobian-cost variant reaches 67.45% mean success rate vs 60.83% for the diffusion baseline and 57.00% for RobotFingerPrint across H→A, H→S, S→A.
- GWH-cost and contact-cost variants reach 14.63% and 14.54% mean 6D GWH IoU, versus 9.58% for the diffusion baseline.
- Contact-based OT variant achieves 4.99 cm^2 mean contact alignment error, the lowest among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Jacobian OT cost instead of the other tested OT costs | Grasp2Grasp · Human→Allegro, Human→Shadow, Shadow→Allegro (mean) | success rate (%): 67.45% and 66.34% | Helped |
| Use contact map OT cost instead of other tested OT costs | Grasp2Grasp · Human→Allegro, Human→Shadow, Shadow→Allegro (mean) | diversity (rad): 0.272 | Helped |
| Use GWH or contact map OT cost instead of other tested OT costs | Grasp2Grasp · Human→Allegro, Human→Shadow, Shadow→Allegro (mean) | 6D GWH IoU (%): 14.63% and 14.54% | Helped |
| Set diffusion rate σ to 0.01 rather than 0.1 (ablation) | Contact OT model · Human→Allegro (H→A) | success rate (%): 74.78 -> 77.16 | Helped |
| Set diffusion rate σ to 1.0 rather than 0.1 (ablation) | Contact OT model · Human→Allegro (H→A) | success rate (%): 74.78 -> 64.39 | Hurt |
| Use 100 rather than 10 integration steps (ablation) | Grasp2Grasp (Contact OT) · Human→Allegro (H→A) | success rate (%): 71.45 -> 75.12 | Helped |
| Increase integration steps from 100 to 200 (ablation) | Grasp2Grasp (Contact OT) · Human→Allegro (H→A) | 6D GWH IoU (%): 14.00 -> 15.89 | Helped |
| Remove mesh vertex loss (ablation) | Grasp2Grasp VAE · Held-out objects (Allegro, Shadow, Human) | Hand Parameters L2 Error: 0.0335 -> 0.0967; 0.0286 -> 0.0933; 0.0523 -> 0.1270 | Hurt |
| Remove direct parameter reconstruction loss (ablation) | Grasp2Grasp VAE · Held-out objects (Allegro, Shadow, Human) | Hand Parameters L2 Error: 0.0335 -> 0.0439; 0.0286 -> 0.0441; 0.0523 -> 0.0534 | Hurt |
| Replace proposed VAE with PointNet VAE (ablation) | Grasp2Grasp VAE · Held-out objects (Allegro, Shadow, Human) | Hand Parameters L2 Error: 0.0335 -> 0.1965; 0.0286 -> 0.2001; 0.0523 -> 0.2468 | Hurt |
Surprising findings
- Ours with diffusion rate sigma=1.0 yields higher diversity but lower success and IoU, indicating noisy reference dynamics hurt physical precision.
- The diffusion baseline degrades sharply at 10 integration steps (60.00% success) while the SB model keeps 71.45%.
Limitations
- Performance drops for thin-shell objects with minimal contact regions.
- Shadow hand results are consistently lower than Allegro across metrics.
- Does not generalize to unseen hand morphologies at inference; the VAE decoder needs retraining or fine-tuning for a new hand.
Experimental setup
Datasets: MultiGripperGrasp ShapeNet
Benchmarks: MultiGripperGrasp
Metrics: grasp success rate (IsaacGym perturbation test) diversity (rad) 6D GWH IoU contact alignment (cm^2) inference time
Baselines: RobotFingerPrint Dex-Retargeting CrossDex Diffusion
Builds on: [SF]^2M Schrödinger Bridge entropic optimal transport GenDexGrasp RobotFingerPrint
Closest papers by meaning
- Contact Map Transfer with Conditional Diffusion Model for Generalizable Dexterous Grasp Generation (NeurIPS 2025 · similarity 0.89)
- Scaffolding Dexterous Manipulation with Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation (NeurIPS 2026 · similarity 0.87)
- LatentHOI: On the Generalizable Hand Object Motion Generation with Latent Hand Diffusion. (CVPR 2025 · similarity 0.87)
- UniGraspTransformer: Simplified Policy Distillation for Scalable Dexterous Robotic Grasping (CVPR 2025 · similarity 0.87)
- Cross-Embodiment Dexterous Grasping with Reinforcement Learning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.