Research map / Vision-language and multimodal / Robot action and navigation
Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Robot-R1 trains Qwen2.5-VL-7B with GRPO on multiple-choice next-keypoint, current-state, and movement QA built from RLBench expert demos, and introduces an open-ended Robot-R1 Bench for embodied reasoning.
Problem
LVLMs lack the embodied reasoning needed for robot control, and SFT on heuristically built embodied QA datasets is poorly aligned with low-level control and causes catastrophic forgetting and weak generalization.
Approach
Reformulate next-keypoint state prediction as a multiple-choice QA with auxiliary current-state and movement QA tasks, and optimize the model's explicit reasoning with GRPO using answer-correctness rewards, instead of SFT.
Key results
- On Robot-R1 Bench low-level control, Robot-R1 reaches 0.76 movement and 1.51 spatial average, beating GPT-4o (0.72 and 1.43) and the Qwen2.5-VL-7B base (0.58 and 1.40).
- On EmbodiedBench Manipulation, Robot-R1 averages 11.68% success versus 8.92% for Qwen2.5-VL-7B-Ins, while Direct SFT and CoT SFT score 0%.
- On SpatialRGPT-Bench, Robot-R1 reaches 15.89 quantitative average accuracy versus 9.88 for the base model, and 40.79 qualitative average versus 29.07.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with Robot-R1 reinforcement learning instead of leaving the base model unfine-tuned | Qwen2.5-VL-7B-Ins · Robot-R1 Bench, Movement Avg | score: 0.58 -> 0.76 | Helped |
| Train with Robot-R1 reinforcement learning instead of leaving the base model unfine-tuned | Qwen2.5-VL-7B-Ins · Robot-R1 Bench, Planning | score: 1.66 -> 1.44 | Hurt |
| Train with Robot-R1 instead of CoT SFT | Qwen2.5-VL-7B-Ins · EmbodiedBench Manipulation | Avg. success rate (%): 0 -> 11.68 | Helped |
| Train with Robot-R1 instead of the base model | Qwen2.5-VL-7B-Ins · SpatialRGPT-Bench, quantitative tasks | Quant. Avg accuracy (%): 9.88 -> 15.89 | Helped |
| Train with Robot-R1 instead of the base model | Qwen2.5-VL-7B-Ins · SpatialRGPT-Bench, qualitative tasks | Qual. Avg accuracy (%): 29.07 -> 40.79 | Helped |
| Use RLOO instead of GRPO (ablation) | Robot-R1 · Robot-R1 Bench, High-level Action | score: 1.30 -> 1.54 | Helped |
| Use RLOO instead of GRPO (ablation) | Robot-R1 · Robot-R1 Bench, Movement | score: 0.76 -> 0.68 | Hurt |
| Use REINFORCE++ instead of GRPO (ablation) | Robot-R1 · Robot-R1 Bench, Movement | score: 0.76 -> 0.62 | Hurt |
| Replace MCQA with open-ended waypoint generation (ablation) | Robot-R1 · Robot-R1 Bench, Movement | score: 0.76 -> 0.68 | Hurt |
| Add current-state prediction QA to open-ended waypoint prediction (ablation) | Robot-R1 · Robot-R1 Bench, Movement | score: 0.68 -> 0.48 | Hurt |
| Add movement prediction QA to MCQA waypoint and current-state prediction (ablation) | Robot-R1 · Robot-R1 Bench, Movement | score: 0.54 -> 0.76 | Helped |
| Add current-state prediction QA to MCQA waypoint prediction (ablation) | Robot-R1 · Robot-R1 Bench, Spatial | score: 1.42 -> 1.51 | Helped |
Surprising findings
- Reasoning length decreases during RL training, the opposite of the trend typically reported for DeepSeek-R1 on math and coding.
- Planning performance slightly decreases after Robot-R1 training (1.66 base to 1.44), which the authors attribute to the keypoint-focused objective.
Limitations
- Only xyz end-effector position is modeled; rotation and gripper state are excluded, limiting reasoning on complex tasks.
- Training uses only five RLBench tasks with limited object diversity, which the authors say hurts generalization (e.g., LIBERO Objective drop).
- The Robot-R1 Bench is small (215 questions) and uses GPT-4o as judge, with a low human correlation for planning (0.33).
Experimental setup
Datasets: RLBench demonstrations (5 tasks, 50 demos each) EmbodiedBench Manipulation SpatialRGPT-Bench VLABench LIBERO BridgeV2
Benchmarks: Robot-R1 Bench EmbodiedBench Manipulation SpatialRGPT-Bench VLABench LIBERO Robot-R1 Bridge Bench
Metrics: GPT-4o judge score (0-3) success rate (%) accuracy (%) Pearson correlation with human scores
Baselines: Direct SFT CoT SFT GPT-4o Claude-3.7-Sonnet Gemini-2.0-Flash RLOO REINFORCE++
Builds on: DeepSeek-R1 GRPO ARM
Closest papers by meaning
- SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models (ICML 2024 · similarity 0.90)
- LaST-R1: Reinforcing Robotic Manipulation via Adaptive Physical Latent Reasoning (NeurIPS 2026 · similarity 0.90)
- RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics (NeurIPS 2025 · similarity 0.89)
- RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models (NeurIPS 2026 · similarity 0.89)
- Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.