Research map / Image generation / Autoregressive and latent generation
Language-driven Grasp Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Grasp-Anything++, a synthetic 1M-image language-grasp dataset built with foundation models, and LGD, a diffusion-based language-conditioned grasp detector trained with an added contrastive loss on the denoising objective.
Problem
Language-driven grasp detection, where a robot must produce grasp poses for an object or object part specified by a natural-language instruction, lacks large-scale language-annotated grasp data and methods that fuse vision and text effectively.
Approach
Treats language-conditioned grasp detection as conditional generation with a diffusion model, adding a contrastive loss (derived from a variational bound) that pulls the predicted clean grasp estimate toward the ground truth during denoising, with ALBEF fusion of image and text features.
Key results
- LGD with CLIP text encoder reaches 0.48 seen, 0.42 unseen, H=0.45 on Grasp-Anything++, versus 0.33 H for the best baseline CLIP-Fusion.
- On real robot (KUKA LBR iiwa), LGD achieves 0.43 single-object and 0.42 cluttered success, above CLIP-Fusion's 0.40 and 0.40.
- Training GR-ConvNet on Grasp-Anything++ yields 0.41 to 0.65 cross-dataset accuracy, improving over other training datasets by roughly 10-33 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LGD with CLIP instead of CLIP-Fusion | Language-driven grasp detection · Grasp-Anything++ (H) | success rate: 0.33 -> 0.45 | Helped |
| Use LGD instead of CLIP-Fusion | Robotic grasping · Single-object robotic grasping | success rate: 0.40 -> 0.43 | Helped |
| Use LGD instead of CLIP-Fusion | Robotic grasping · Cluttered-scene robotic grasping | success rate: 0.40 -> 0.42 | Helped |
| Remove text input from LGD (ablation) | LGD · Cornell (H) | success rate: 0.86 -> 0.85 | Hurt |
| Remove text input from LGD (ablation) | LGD · Grasp-Anything++, Jacquard, VMRD, and OCID-grasp (H) | success rate: 0.64 -> 0.68; 0.75 -> 0.77; 0.70 -> 0.72; 0.75 -> 0.76 | Helped |
| Do not predict x̃0 (ablation) | LGD · Grasp-Anything++ (H) | success rate: 0.10 -> 0.45 | Hurt |
| Add contrastive loss (ablation) | LGD · Grasp-Anything++ (H) | success rate: 0.42 -> 0.45 | Helped |
Surprising findings
- Language-conditioned grasp detection on Grasp-Anything++ is harder than standard grasp detection; the same approaches score lower on Grasp-Anything++ than on Jacquard, Cornell, VMRD, and OCID-grasp.
- A model trained only on synthetic Grasp-Anything++ images generalizes to real-world images and robot grasping.
Limitations
- Dataset lacks depth images, limiting direct use on robots.
- Dataset creation is time-consuming and depends on ChatGPT API access.
- Manual filtering of hallucinated Stable Diffusion images introduces possible bias.
Experimental setup
Datasets: Grasp-Anything++ Jacquard Cornell VMRD OCID-grasp LVIS
Benchmarks: Grasp-Anything++ Jacquard Cornell VMRD OCID-grasp
Metrics: success rate (IoU>25% and angle offset <30°) harmonic mean (H)
Baselines: GR-ConvNet + CLIP Det-Seg-Refine + CLIP GG-CNN + CLIP CLIPORT CLIP-Fusion LGD w/o contrastive loss
Builds on: denoising diffusion probabilistic models ALBEF Grasp-Anything GR-ConvNet CLIPORT
Closest papers by meaning
- RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulation (CVPR 2026 · similarity 0.89)
- Grasp as You Say: Language-guided Dexterous Grasp Generation (NeurIPS 2024 · similarity 0.89)
- Grasp2Grasp: Vision-Based Dexterous Grasp Translation via Schrödinger Bridges (NeurIPS 2025 · similarity 0.86)
- DexGrasp Anything: Towards Universal Robotic Dexterous Grasping with Physics Awareness (CVPR 2025 · similarity 0.86)
- LatentHOI: On the Generalizable Hand Object Motion Generation with Latent Hand Diffusion. (CVPR 2025 · similarity 0.85)
- Active Perception for Grasp Detection via Neural Graspness Field (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.