Research map / Vision-language and multimodal / Robot action and navigation

RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation

NeurIPS 2026 · Vision-language and multimodal › Robot action and navigation · approach: Vision-language-action foundation models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces RoboProcessBench, a 12-family multiple-choice VQA benchmark and ~58k-item ProcessData corpus for evaluating and post-training VLMs on process-aware understanding of robotic manipulation, showing zero-shot weaknesses on progress and temporal tasks and gains from LoRA SFT on Qwen2.5-VL-7B and InternVL-3-8B.

Problem

Existing evaluations do not test whether VLMs understand fine-grained process progression (contact, motion, local progress, temporal order, primitive transitions) in robotic manipulation, which is needed for VLMs acting as visual critics, reward generators, and failure detectors.

Approach

Decomposes process-aware manipulation understanding into 12 diagnostic question families over local process units anchored to contact or primitive annotations, builds QA items from physically grounded execution traces across four robot datasets, and shows the same data can serve as SFT supervision that transfers across two VLM backbones.

Key results

• ProcessData-SFT-Qwen reaches 92.5% on T10 and 96.5% on T11, versus at most 58.0% for zero-shot Qwen2.5-VL-7B on T6 and 33.1% on T11 for its base.
• Best zero-shot T5 (primitive-local progress) accuracy is only 34.4%, near the 33.3% chance baseline; post-trained models reach about 45%.
• Best zero-shot T8 (temporal ordering) accuracy is 22.3%; post-training leaves T8 near chance (about 16-17%).

Surprising findings

• Temporal ordering (T8) stays near chance even after supervised fine-tuning on both backbones.
• Zero-shot models score near chance on primitive-local progress (T5), with the best at 34.4% against a 33.3% chance rate.

Limitations

• Multiple-choice QA scores do not directly indicate closed-loop policy success or deployable reward quality.
• Evaluates structured visual QA rather than open-ended reasoning or long-horizon video memory.
• Limited to the four source datasets and their viewpoints; failure-rich trajectories not covered.

Experimental setup

Datasets: GM-100 RH20T REASSEMBLE AIST-Bimanual

Benchmarks: RoboProcessBench (ProcessData-Eval)

Metrics: accuracy

Baselines: Qwen2.5-VL-7B Qwen3-VL-8B Qwen3-VL-32B InternVL-3-8B InternVL-3.5-8B InternVL-3-38B InternVL-3.5-38B RoboBrain-2.0-7B GLM-4.6V Gemini-3.1-Flash GPT-4o GPT-5.4-mini

Builds on: PRM-as-a-Judge RoboPulse

Closest papers by meaning

• AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation (ICLR 2025 · similarity 0.91)
• RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation (NeurIPS 2025 · similarity 0.90)
• TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies (ICLR 2025 · similarity 0.89)
• Robotic Visual Instruction (CVPR 2025 · similarity 0.89)
• SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation (ICML 2025 · similarity 0.89)
• Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization (NeurIPS 2025 · similarity 0.89)