Research map / Vision-language and multimodal / Robot action and navigation
ManipEvalAgent: Promptable and Efficient Evaluation Framework for Robotic Manipulation Policies
Yiteng Chen, Huiping Zhuang, WenboLi, Shiyi Wang, Xiangyu Zhao, Qingyao Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace fixed-benchmark evaluation with small-batch multi-round ManipEvalAgent | policy evaluation (ACT) · RoboTwin single-task | evaluation time (min): 167 min -> 42 min | Improved |
| use ManipEvalAgent in multi-task setting instead of full benchmarks | RDT evaluation on RoboTwin · multi-task | evaluation time (min): 11102 min -> 97 min | Improved |
| ManipEvalAgent conclusions vs full benchmark success rate | RDT, single-task RoboTwin · RoboTwin success rate | trials within error margin: 60% (RDT) to 100% (DP, pi0) | Mixed |
| add retrieval-augmented generation to TaskGen (ablation) | TaskGen · task code generation | success rate (%): 95% -> 98% | Improved |
| add visual self-check to TaskGen (ablation) | TaskGen · task code generation | success rate (%): 96% -> 98% | Improved |
| add README.Agent to TaskGen (ablation) | TaskGen · task code generation | success rate (%): 96% -> 98% | Improved |
| add RAG to ToolGen (ablation) | ToolGen · tool code generation | success rate (%): 92% -> 96% | Improved |
| add scene perturbations (clutter) to VQA input (ablation) | VLM-based VQA tools · RoboTwin 2.0 VQA accuracy | accuracy: 0.997 -> 0.989 (GPT-4o) | No clear effect |
| increase rollouts from 10 to 20/50 for ranking consistency (ablation) | ManipEvalAgent · policy ranking Spearman rho | Spearman rho: 0.83 -> 0.79 -> 0.81 (RoboTwin short) | No clear effect |
Closest papers by meaning
- RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation (NeurIPS 2025 · similarity 0.88)
- GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation (CVPR 2025 · similarity 0.88)
- RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation (NeurIPS 2026 · similarity 0.87)
- CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation (ICML 2026 · similarity 0.87)
- RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills (NeurIPS 2025 · similarity 0.87)
- DexFlyWheel: A Scalable and Self-improving Data Generation Framework for Dexterous Manipulation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
