Research map / Vision-language and multimodal / Robot action and navigation
RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ShareRobot, a ~1M QA robotic dataset labeled for planning, affordance, and trajectory, and RoboBrain, a LLaVA-style MLLM (Qwen2.5-7B + SigLIP) trained in multi-stage fashion with LoRA adapters for affordance and trajectory.
Problem
Multimodal LLMs lack the planning, affordance perception, and trajectory prediction capabilities needed for long-horizon robotic manipulation, largely due to a scarcity of fine-grained robotic datasets.
Approach
Build a large fine-grained dataset (ShareRobot) from Open X-Embodiment with planning QA, affordance boxes, and 2D end-effector trajectories, then train a unified MLLM with a general-then-robotic multi-stage curriculum plus task-specific LoRA modules for affordance (A-LoRA) and trajectory (T-LoRA).
Key results
- On RoboVQA, RoboBrain's BLEU-4 exceeded the second-best model by 18.75.
- Affordance AP on AGD20K reached 27.1%, versus 12.5% for Qwen2-VL-7B and 9.8% for LLaVA-NeXT-7B.
- Trajectory variant with all design choices cut DFD, HD, and RMSE by 42.9%, 94.2%, and 31.6% relative to the baseline variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RoboBrain instead of Qwen2-VL-7B | affordance prediction · AGD20K affordance test set | AP: 12.5 % -> 27.1 % | Helped |
| use RoboBrain instead of LLaVA-NeXT-7B | affordance prediction · AGD20K affordance test set | AP: 9.8 % -> 27.1 % | Helped |
| add end-effector 2D start coordinates (ablation) | RoboBrain trajectory prediction · Trajectory prediction | DFD: 0.191 -> 0.176 | Helped |
| add end-effector 2D start coordinates (ablation) | RoboBrain trajectory prediction · Trajectory prediction | HD: 0.171 -> 0.157 | Helped |
| add end-effector 2D start coordinates (ablation) | RoboBrain trajectory prediction · Trajectory prediction | RMSE: 0.133 -> 0.117 | Helped |
| limit waypoints to 10 via uniform sampling (ablation) | RoboBrain trajectory prediction with start points · Trajectory prediction | DFD: 0.176 -> 0.185 | Hurt |
| limit waypoints to 10 via uniform sampling (ablation) | RoboBrain trajectory prediction with start points · Trajectory prediction | HD: 0.157 -> 0.163 | Hurt |
| limit waypoints to 10 via uniform sampling (ablation) | RoboBrain trajectory prediction with start points · Trajectory prediction | RMSE: 0.117 -> 0.125 | Hurt |
| add end-effector positions and special tokens for waypoints and start/goal points (ablation) | RoboBrain trajectory prediction with start points and max points · Trajectory prediction | DFD: 0.185 -> 0.109 (42.9%↓) | Helped |
| add end-effector positions and special tokens for waypoints and start/goal points (ablation) | RoboBrain trajectory prediction with start points and max points · Trajectory prediction | HD: 0.163 -> 0.010 (94.2%↓) | Helped |
| add end-effector positions and special tokens for waypoints and start/goal points (ablation) | RoboBrain trajectory prediction with start points and max points · Trajectory prediction | RMSE: 0.125 -> 0.091 (31.6%↓) | Helped |
Limitations
- Trajectory and affordance heads are LoRA modules rather than a jointly trained unified output, and the planning, affordance, and trajectory evaluations are on separate test splits.
- Planning quality is scored with GPT-4o as judge, which may introduce evaluator bias.
- Dataset labels for planning are generated by Gemini and reviewed by three annotators, so label noise is possible.
Experimental setup
Datasets: ShareRobot Open X-Embodiment RoboVQA LLaVA-OneVision-Data LCS-558K ScanView/MMScan 3RScan ScanQA SQA3D AGD20K
Benchmarks: RoboVQA OpenEQA ShareRobot test set AGD20K trajectory prediction test set
Metrics: BLEU-1 to BLEU-4 GPT-4o-judged score Average Precision (AP) Discrete Fréchet Distance Hausdorff Distance RMSE
Baselines: GPT-4V Claude3 LLaVA-1.5 LLaVA-OneVision-7b Qwen2-VL-7b RoboMamba LLaVA-NeXT-7B
Builds on: LLaVA LLaVA-OneVision Open X-Embodiment RoboVQA RoboMamba
Closest papers by meaning
- RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation (NeurIPS 2025 · similarity 0.88)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.88)
- ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation (CVPR 2024 · similarity 0.88)
- RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation (NeurIPS 2026 · similarity 0.88)
- RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models (ICML 2024 · similarity 0.88)
- Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.