Research map / Vision-language and multimodal / Robot action and navigation
RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RoboCerebra, a LIBERO-based long-horizon manipulation benchmark with GPT-generated tasks, human simulated demonstrations, and a hierarchical VLM-planner plus OpenVLA-controller framework evaluated on System 2 planning, reflection, and memory.
Problem
Existing robotic manipulation benchmarks are too short and structurally simple to evaluate System 2 abilities (long-horizon planning, reflection, memory) of VLMs used as high-level planners.
Approach
A top-down pipeline where GPT-o3-mini generates high-level tasks and decomposes them into subtasks, humans execute them in LIBERO simulation with dynamic object variations, and a hierarchical VLM planner with memory bank drives a fine-tuned OpenVLA controller; evaluation covers planning, reflection, and memory via multi-dimensional metrics.
Key results
- Hierarchical Framework (GPT-4o planner + fine-tuned OpenVLA) reaches 16.55% average success vs 4.57% for fine-tuned OpenVLA alone and 2.00% for OpenVLA-Libero100.
- GPT-4o is the best System 2 planner with 16.04% average success and 68.33% planning accuracy, outperforming Qwen2.5-VL-7B (11.19%) and LLaVA-Next-Video (11.37%).
- Ground-truth plan execution reaches 25.16% average success, leaving a gap of over 9 points to GPT-4o.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add hierarchical monitoring and iterative plan updates instead of using a one-shot planner | Planner+OpenVLA* · RoboCerebra, Mix | success rate (%): 11.48 -> 13.21 | Helped |
| Add hierarchical monitoring and iterative plan updates instead of using a one-shot planner | Planner+OpenVLA* · RoboCerebra, Ideal | success rate (%): 21.92 -> 21.10 | Hurt |
| Fine-tune OpenVLA on RoboCerebra | OpenVLA · RoboCerebra, Ideal | success rate (%): 4.05 -> 7.84 | Helped |
| Add a System 2 planner | OpenVLA* · RoboCerebra, Mix | success rate (%): 0.00 -> 11.48 | Helped |
| Disable visual input to GPT-4o (ablation) | GPT-4o planner with OpenVLA* · RoboCerebra, Avg | success rate (%): 16.04 -> 15.10 | Hurt |
| Use GPT-4o instead of Qwen2.5-VL (ablation) | System 2 planner with OpenVLA* · RoboCerebra, Avg | success rate (%): 11.19 -> 16.04 | Helped |
| Use GPT-4o instead of LLaVA-Next-Video (ablation) | System 2 planner with OpenVLA* · RoboCerebra, Avg | success rate (%): 11.37 -> 16.04 | Helped |
| Fine-tune Qwen2.5-VL-7B on the video-instruction dataset (ablation) | Qwen2.5-VL-7B System 2 planner · RoboCerebra, reflection | Acc_C: 47.74 -> 66.83 | Helped |
| Fine-tune Qwen2.5-VL-7B on the video-instruction dataset (ablation) | Qwen2.5-VL-7B System 2 planner · RoboCerebra, task success | SR: 11.19 -> 9.33 | Hurt |
| Replace OpenVLA with π0-fast (ablation) | GPT-4o System 2 planner · RoboCerebra, Avg | success rate (%): 16.04 -> 15.15 | No clear effect |
| Replace OpenVLA with π0-fast (ablation) | Qwen2.5-VL System 2 planner · RoboCerebra, Avg | success rate (%): 11.19 -> 13.19 | Mixed |
| Replace OpenVLA with π0-fast (ablation) | GPT-4o-Blind System 2 planner · RoboCerebra, Avg | success rate (%): 15.10 -> 13.63 | Hurt |
Surprising findings
- GPT-4o without visual input (GPT-4o-Blind, 15.10%) performs nearly as well as GPT-4o with vision (16.04%), suggesting environment observation has not yet played a decisive role.
- Fine-tuned Qwen2.5-VL-7B-SFT achieves the highest reflection accuracy (66.83%) but a lower task success rate (9.33%) than the base models.
Limitations
- Interaction between System 1 and System 2 is limited to one-way subgoal communication.
- Evaluation is simulation-only; real-world deployment is untested.
- Overall success rates remain low (best around 16%), indicating the benchmark is far from solved.
Experimental setup
Datasets: RoboCerebra training set LIBERO
Benchmarks: RoboCerebra
Metrics: task success rate planning accuracy plan efficiency action completion accuracy average plan length
Baselines: OpenVLA-Libero100 OpenVLA* Planner+OpenVLA* GPT-4o-Blind Qwen2.5-VL LLaVA-Next-Video Cosmos-Reason1 VeBrain RoboBrain-2.0 pi0-fast
Builds on: OpenVLA LIBERO
Closest papers by meaning
- RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulation (NeurIPS 2024 · similarity 0.90)
- RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation (NeurIPS 2026 · similarity 0.90)
- Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulation (AAAI 2025 · similarity 0.90)
- RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models (ICML 2024 · similarity 0.90)
- Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks (ICLR 2024 · similarity 0.89)
- UniDomain: Pretraining a Unified PDDL Domain from Real-World Demonstrations for Generalizable Robot Task Planning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.