Research map / Vision-language and multimodal / Robot action and navigation
VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VLA-REPLICA, a ~$1050 reproducible real-world benchmark (SO-101 arm, light box, overlay-based scene reproduction) with 10 tasks, 500 demos, and 90 ID/OOD test scenes for evaluating imitation learning and VLA policies.
Problem
Real-world evaluation of VLA models lacks accessible, reproducible, and consistent benchmarks; existing real-world benchmarks are expensive, centralized, or narrow in task diversity.
Approach
A low-cost, locally executable real-world benchmark that standardizes hardware (SO-101 arm, light box, AprilTag and visual overlay camera calibration), a universal joint-space action normalization, and reference-image-based test scene reconstruction, with ID and OOD evaluation protocols.
Key results
- pi0.5 achieves the highest average ID success rate of 0.54 across 10 tasks, versus 0.34 for pi0 and 0.18 for ACT.
- On OOD tasks, pi0.5 averages 0.35 success, while all methods drop sharply on memory-counting tasks (e.g., press button n times at 0 for all models).
- A second independently built setup yields success rates comparable to the original platform across models and selected tasks.
Surprising findings
- None of the evaluated policies, including VLAs, generalize to memory-counting tasks; policies keep repeating actions rather than tracking counts.
- Rigid-object tasks such as stacking blocks and putting blocks in a box have low success even for the strongest models, despite good performance on deformable-object tasks.
Limitations
- Focuses on tabletop manipulation with a single low-cost robot embodiment (SO-101).
- Task scale is smaller than large-scale simulation benchmarks.
- Small hardware, lighting, and object placement differences may still cause inconsistencies across setups.
Experimental setup
Datasets: VLA-REPLICA demonstrations (500 teleoperated SO-101 demonstrations)
Benchmarks: VLA-REPLICA ID tasks (50 test scenes) VLA-REPLICA OOD tasks (40 test scenes)
Metrics: success rate
Baselines: ACT DiT-D DiT-F SmolVLA X-VLA
Builds on: LeRobot Diffusion Policy ACT
Closest papers by meaning
- NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.93)
- VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Model (ICML 2026 · similarity 0.93)
- ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.92)
- VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Caching (NeurIPS 2025 · similarity 0.92)
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.92)
- CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (CVPR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.