Research map / Large language models / Agents, code and math
CheckManual: A New Challenge and Benchmark for Manual-based Appliance Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CheckManual, a CAD-aligned appliance manual dataset and simulated benchmark with three manipulation tracks, plus the ManualPlan baseline that plans from multi-page manuals.
Problem
Robots cannot use electrical appliances correctly because they lack function information that only appliance manuals provide; prior manual work is limited to QA and lacks CAD models for manipulation evaluation.
Approach
Generate multi-page appliance manuals from PartNet-Mobility CAD models using GPT-4o for annotation and LaTeX generation with human verification, then define three manual-based manipulation tracks evaluated in SAPIEN, and a ManualPlan planner that resolves manuals, plans steps with an LLM, and aligns parts via Set-of-Mark.
Key results
- ManualPlan achieves 22.22% part alignment / 22.22% task planning in Track 1 total ... reported as 22.96 / 22.22 for alignment/planning on the total column in Track 1 with 27.32 / 20.70 overall
- Removing the manual drops Track 1 total planning success from 20.70 to 7.99 (w/o manual in Track 1 total column)
- Track 3 ManualPlan with VoxPoser reaches a total task completion rate of 5.18 and task success rate of 2.87, showing long-horizon manual-based manipulation remains very hard
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add appliance manual input (ablation) | ManualPlan · CheckManual Track 1, Total | task planning success rate: 7.99 -> 20.70 | Helped |
| add appliance manual input (ablation) | ManualPlan + Primitive Actions · CheckManual Track 2, Total | task completion rate: 2.47 -> 4.50 | Helped |
| add appliance manual input (ablation) | ManualPlan + Primitive Actions · CheckManual Track 2, Total | task success rate: 1.37 -> 2.39 | Helped |
| add appliance manual input (ablation) | ManualPlan + VoxPoser · CheckManual Track 3, Total | task completion rate: 1.11 -> 5.18 | Helped |
| add appliance manual input (ablation) | ManualPlan + VoxPoser · CheckManual Track 3, Total | task success rate: 0.55 -> 2.87 | Helped |
Surprising findings
- Even state-of-the-art MLLMs struggle to align manual part annotations with CAD model parts, limiting planning performance
- Track 2 and Track 3 execution success rates are very low, with errors accumulating across long-horizon steps
Limitations
- Manuals are synthetic and generated from CAD models, not real manuals
- Low execution success in CAD-free and pure manual tracks
- Simulation-only evaluation apart from a brief real robot deployment
Experimental setup
Datasets: PartNet-Mobility
Benchmarks: CheckManual
Metrics: part alignment success rate task planning success rate task completion rate task success rate
Baselines: VoxPoser ManualPlan w/o manual
Builds on: Set-of-Mark VoxPoser PartNet-Mobility FoundationPose AnyGrasp
Closest papers by meaning
- RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills (NeurIPS 2025 · similarity 0.82)
- HumanoidGen: Data Generation for Bimanual Dexterous Manipulation via LLM Reasoning (NeurIPS 2025 · similarity 0.82)
- RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation (NeurIPS 2025 · similarity 0.82)
- ManiSkill-HAB: A Benchmark for Low-Level Manipulation in Home Rearrangement Tasks (ICLR 2025 · similarity 0.82)
- RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation (NeurIPS 2026 · similarity 0.81)
- AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation (ICLR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.