Research map / Vision-language and multimodal / Robot action and navigation
VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks
Shiduo Zhang, Zhe Xu, Peiju Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin, Zuxuan Wu, Yu-Gang Jiang, Xipeng Qiu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add VLA pretraining (vs from scratch) (ablation) | OpenVLA · VLABench primitive, seen | Progress Score: 3.02 -> 11.74 | Improved |
| add VLA pretraining (vs from scratch) (ablation) | RDT-1B · VLABench primitive, seen | Progress Score: 6.26 -> 15.37 | Improved |
| add VLA pretraining (vs from scratch) (ablation) | Octo · VLABench primitive, seen | Progress Score: 1.02 -> 1.34 | Improved |
| use closed-loop control instead of open-loop (64 steps) (ablation) | RDT-1B · VLABench primitive, seen | Progress Score: 15.37 -> 17.68 | Improved |
| shorten open-loop horizon from 64 to 32 steps (ablation) | RDT-1B · VLABench primitive, seen | Progress Score: 15.37 -> 15.52 | No clear effect |
| scale fine-tuning data from 100 to 2000 samples (ablation) | Octo, OpenVLA, RDT-1B · Put Box on Painting | Progress Score | No clear effect |
| evaluate on unseen object categories instead of seen | RDT-1B · VLABench primitive | Progress Score: 15.37 -> 9.08 | Hurt |
| evaluate on unseen object categories instead of seen | OpenVLA · VLABench primitive | Progress Score: 11.74 -> 7.93 | Hurt |
| switch to common-sense instructions for unseen objects | RDT-1B · VLABench primitive, unseen | Progress Score: 9.08 -> 3.61 | Hurt |
| switch to common-sense instructions for unseen objects | OpenVLA · VLABench primitive, unseen | Progress Score: 7.93 -> 5.11 | Hurt |
| add visual perception (GPT-4V) to Voxposer (ablation) | Voxposer · VLABench workflows | Progress Score | Mixed |
Closest papers by meaning
- VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.92)
- RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation (NeurIPS 2025 · similarity 0.91)
- UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent (ICML 2025 · similarity 0.91)
- NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.91)
- CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (CVPR 2025 · similarity 0.91)
- Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
