Research map / Vision-language and multimodal / Multimodal LLMs

RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces RealVLG-11B, a real-world visual-language grounding and grasping dataset built by extending existing grasp datasets with LVLM-generated and human-verified language, and RealVLG-R1, a Qwen2.5-VL model trained with GRPO/GSPO verifiable rewards to predict boxes, masks, grasp rectangles, and contact points.

Problem

Visual-language grounding focuses on coarse object-level localization and lacks actionable grasp reasoning, while grasping datasets lack fine-grained language annotations and existing language-grasp datasets are synthetic, low-resolution, and have noisy grasp labels.

Approach

Unify bounding box, segmentation, grasp rectangle, and contact point prediction in one LVLM trained with reinforcement fine-tuning (GRPO/GSPO) using task-specific verifiable IoU/S-measure/Huber/contact rewards, on a real-world dataset with language descriptions verified by VLMs and humans.

Key results

• RealVLG-R1-3B (GRPO) reaches 87.2 gIoU on the Seen bbox task and 34.7 mIoU grasp on Seen, versus 4.3 mIoU for Qwen2.5-VL-3B without task training.
• RealVLG-R1-7B (GSPO) achieves 88.5 gIoU on the Novel split with 100% validity rate.
• On a real Franka robot, RealVLG-R1 averages 81% success in the Single setting versus 38% for GraspNet, and 79% in the Clutter setting versus 2% for LGD.

Surprising findings

• Qwen2.5-VL with SFT gives only marginal gains over the base model, while RL yields large grasp improvements (over 30 points mIoU on Seen for 3B).

Limitations

• Grasp predictions are 2D rectangles or contact points, not full 6-DoF poses; 3D extension left to future work.
• Closed-source and open baselines have low validity or grasp accuracy; real-robot evaluation uses only ten objects with 10 trials each.
• Language annotations depend on GPT-4o and Qwen-VL-Max outputs, with human verification only partly described.

Experimental setup

Datasets: RealVLG-11B Cornell VMRD OCID-Grasp GraspNet GraspClutter6D Grasp-Anything Grasp-Anything++

Benchmarks: RealVLG-11B Seen split RealVLG-11B Similar split RealVLG-11B Novel split

Metrics: gIoU cIoU S-measure F-measure mIoU grasp accuracy validity rate real-robot grasp success rate

Baselines: Qwen-VL-Max Gemini2.5-Flash Qwen2.5-VL-3B Qwen2.5-VL-7B Qwen2.5-VL+SFT GraspNet LGD

Builds on: DeepSeek-R1 GRPO GSPO SAM2 Grasp-Anything

Closest papers by meaning

• Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization (NeurIPS 2025 · similarity 0.89)
• Language-driven Grasp Detection (CVPR 2024 · similarity 0.89)
• RoboGround: Robotic Manipulation with Grounded Vision-Language Priors (CVPR 2025 · similarity 0.89)
• Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation (CVPR 2025 · similarity 0.89)
• Localizing, Structuring, and Rendering: Bridging 3D and 2D Vision-Language-Action Models for Robotic Manipulation (CVPR 2026 · similarity 0.89)
• Vision-Language Foundation Models as Effective Robot Imitators (ICLR 2024 · similarity 0.89)