Research map / Vision-language and multimodal / Robot action and navigation
RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RoboCodeX, a multimodal LLM that decomposes instructions via tree-of-thought into object-centric units and generates executable robot control code, trained with a code pre-training dataset and iterative self-updating SFT.
Problem
Translating high-level multimodal understanding into precise, generalizable robotic control behaviors across tasks and robot platforms.
Approach
A BLIP-2-style vision-language model with a tree-of-thought decomposition into object-centric units, each expanded with affordance, grasp preference, articulated-object physical properties, and trajectory planning, expressed as executable robot code; a vision adapter aggregates multi-stage ViT class tokens.
Key results
- On simulated manipulation (50 trials), RoboCodeX achieves 0.80 pick-and-place, 0.84 drawers, 0.74 doors, 0.68 put-object-in-drawer, and 0.64 multi-stage success, versus 0.63/0.68/0.52/0.55/0.50 for GPT-4V.
- On embodied navigation, RoboCodeX reaches 40.0 success / 24.2 SPL on HM3D and 40.3 / 22.0 on HSSD, outperforming L3MVN, Pixel-Nav, and ESC on HM3D.
- On general VQA, RoboCodeX-13B scores 71.5 on LLaVA-Bench (vs 70.7 for LLaVA-1.5-13B) and 31.0 on MM-Vet (vs 35.4 for LLaVA-1.5-13B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RoboCodeX instead of GPT-4V | Robotic manipulation · Pick & Place | success rate: 0.63 -> 0.80 | Helped |
| Use RoboCodeX instead of GPT-4V | Robotic manipulation · Drawers | success rate: 0.68 -> 0.84 | Helped |
| Use RoboCodeX instead of GPT-4V | Robotic manipulation · Doors | success rate: 0.52 -> 0.74 | Helped |
| Use RoboCodeX instead of GPT-4V | Robotic manipulation · Put obj. in drawer | success rate: 0.55 -> 0.68 | Helped |
| Use RoboCodeX instead of GPT-4V | Robotic manipulation · Multi-stage | success rate: 0.50 -> 0.64 | Helped |
| Use RoboCodeX instead of the strongest reported baseline | Embodied navigation · HM3D | Success: 39.2 -> 40.0 | Helped |
| Use RoboCodeX instead of the strongest reported baseline | Embodied navigation · HM3D | SPL: 20.5 -> 24.2 | Helped |
| Use RoboCodeX-13B instead of LLaVA-1.5-13B | General multimodal reasoning · LLaVA-Bench | score: 70.7 -> 71.5 | Helped |
| Use RoboCodeX-13B instead of LLaVA-1.5-13B | General multimodal reasoning · MM-Vet | score: 35.4 -> 31.0 | Hurt |
| Add inferred manipulation preferences rather than selecting AnyGrasp's highest-scoring grasp directly (ablation) | RoboCodeX · Four manipulation task types | success rate | Helped |
| Add the vision adapter (ablation) | RoboCodeX · Manipulation tasks | success rate | Helped |
| Add general VQA data during SFT (ablation) | RoboCodeX · Manipulation tasks | average success rate | Helped |
Surprising findings
- RoboCodeX-13B scores lower than LLaVA-1.5-13B on MM-Vet (31.0 vs 35.4), indicating general VQA ability is not fully retained despite the robotics fine-tuning.
Limitations
- Primary failure modes are suboptimal grasp pose selection and trajectory planning failures in long-horizon tasks.
- Relies on external modules (AnyGrasp, GAMMA, GLIP) and simulator-based data, so performance depends on these components.
- Ablation shows removing general VQA data causes overfitting, indicating sensitivity to data mix.
Experimental setup
Datasets: HM3D Google Scanned Objects YCB OmniObject3D AKB-48 ShareGPT4V SViT LLaVA Visual Instruct 150K RT-1 task types LIBERO task types PartNet-Mobility
Benchmarks: LLaVA-Bench MM-Vet HM3D HSSD
Metrics: success rate SPL LLaVA-Bench score MM-Vet score
Baselines: GPT-4V GPT-4 GPT-3.5 L3MVN Pixel-Nav ESC LLaVA-1.5-13B InstructBLIP-13B BLIP-2
Builds on: BLIP-2 Code-as-Policies L3MVN GAMMA AnyGrasp
Closest papers by meaning
- RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models (ICML 2024 · similarity 0.90)
- CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation (ICML 2026 · similarity 0.88)
- ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation (CVPR 2024 · similarity 0.88)
- RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation (NeurIPS 2025 · similarity 0.87)
- Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planning (NeurIPS 2025 · similarity 0.87)
- ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.