Research map / 3D vision / 3D generation
ZeroGrasp: Zero-Shot Shape Reconstruction Enabled Robotic Grasping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZeroGrasp, an octree-based CVAE that jointly reconstructs objects and predicts 6D grasp poses from single-view RGB-D, with a multi-object encoder, 3D occlusion fields, and reconstruction-based grasp refinement, plus a 1M-image synthetic dataset (ZeroGrasp-11B).
Problem
Robotic grasp detection from partial single-view observations often ignores scene geometry, causing collisions and unstable grasps; reconstruction methods that help are slow, need multi-view input, or are not integrated with grasp prediction.
Approach
Couple a sparse octree CVAE for single-view 3D reconstruction with grasp prediction, adding a transformer over per-object latent features to model inter-object relations, 3D occlusion fields from ray-casting instance masks, and a grasp refinement step that adjusts width/depth to contact points on the predicted shape and filters collisions against the reconstruction.
Key results
- On GraspNet-1Billion RealSense seen split, ZeroGrasp reaches 70.53 AP vs 68.21 for EconomicGrasp (prior best), and 72.43 AP with fine-tuning (Ours+FT).
- Reconstruction on the ReOcS hard split: CD 6.73 mm, F1 84.67 vs OCNN-based baseline 8.69 mm CD and 77.58 F1 (ablation baseline row).
- Real-robot evaluation with Franka Panda over 5 scenes: 75% pick success vs 56.25% for the OCNN baseline, both trained only on synthetic data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the multi-object encoder (ablation) | ZeroGrasp · ReOcS hard | CD: 7.09 -> 6.73 | Helped |
| Add 3D occlusion fields (ablation) | ZeroGrasp · ReOcS hard | CD: 7.54 -> 6.73 | Helped |
| Add CVAE (ablation) | ZeroGrasp · ReOcS hard | CD: 7.67 -> 6.73 | Helped |
| Add the multi-object encoder (ablation) | ZeroGrasp · GraspNet-1B Seen | AP: 69.52 -> 70.53 | Helped |
| Add 3D occlusion fields (ablation) | ZeroGrasp · GraspNet-1B Seen | AP: 67.34 -> 70.53 | Helped |
| Add CVAE (ablation) | ZeroGrasp · GraspNet-1B Seen | AP: 70.23 -> 70.53 | Helped |
| Add contact-based constraints (ablation) | ZeroGrasp · GraspNet-1B Seen | AP: 65.67 -> 70.53 | Helped |
| Add collision detection using reconstructed shapes (ablation) | ZeroGrasp · GraspNet-1B Seen | AP: 49.35 -> 70.53 | Helped |
| Use reconstructed shape rather than a depth map for collision detection (ablation) | ZeroGrasp · GraspNet-1B Seen | AP: 59.93 -> 70.53 | Helped |
| Pre-train on ZeroGrasp-11B and fine-tune on GraspNet-1B | ZeroGrasp · GraspNet-1B Seen | AP: 70.53 -> 72.43 | Helped |
| Pre-train on ZeroGrasp-11B and fine-tune on GraspNet-1B | ZeroGrasp · GraspNet-1B Similar | AP: 62.51 -> 65.45 | Helped |
| Pre-train on ZeroGrasp-11B and fine-tune on GraspNet-1B | ZeroGrasp · GraspNet-1B Novel | AP: 26.46 -> 28.49 | Helped |
Surprising findings
- Reconstruction-based collision detection outperforms depth-map collision detection by a large margin (70.53 vs 59.93 AP on seen split).
- Removing the 3D occlusion fields causes a large grasp AP drop (70.53 to 67.34 seen; 26.46 to 25.00 novel), showing occlusion reasoning matters for grasping.
Limitations
- Does not support incremental or multi-view 3D reconstruction.
- Does not account for placement poses that could leverage predicted reconstructions.
- Grasp refinement and collision checks depend on reconstruction accuracy; real-world gains rely on a small 5-scene robot evaluation.
Experimental setup
Datasets: GraspNet-1Billion ZeroGrasp-11B Objaverse-LVIS
Benchmarks: GraspNet-1Billion ReOcS
Metrics: Chamfer distance F1-score@10mm normal consistency average precision (AP) AP0.4 AP0.8 real-robot pick success rate
Baselines: Minkowski OCNN OctMAE GSNet EconomicGrasp Ma et al. GraspNet HGGD CenterGrasp GPD
Builds on: OctMAE EconomicGrasp GraspNet-1Billion OCNN GSNet
Closest papers by meaning
- DexGrasp Anything: Towards Universal Robotic Dexterous Grasping with Physics Awareness (CVPR 2025 · similarity 0.88)
- CADGrasp: Learning Contact and Collision Aware General Dexterous Grasping in Cluttered Scenes (NeurIPS 2025 · similarity 0.87)
- UniGraspTransformer: Simplified Policy Distillation for Scalable Dexterous Robotic Grasping (CVPR 2025 · similarity 0.86)
- DiffGrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model (AAAI 2025 · similarity 0.86)
- DexFuncGrasp: A Robotic Dexterous Functional Grasp Dataset Constructed from a Cost-Effective Real-Simulation Annotation System (AAAI 2024 · similarity 0.86)
- Grasp2Grasp: Vision-Based Dexterous Grasp Translation via Schrödinger Bridges (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.