Research map / Multi-agent systems and games / Human-AI and goal-conditioned
LabUtopia: High-Fidelity Simulation and Hierarchical Benchmark for Scientific Embodied Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LabUtopia, a simulator (LabSim), procedural scene generator (LabScene), and five-level hierarchical benchmark (LabBench) for scientific lab embodied agents, benchmarking ACT, Diffusion Policy and pi0 imitation policies.
Problem
Embodied agents for laboratory automation lack suitable simulators and benchmarks that model chemical transformations, diverse lab assets, and long-horizon tasks from atomic actions to mobile manipulation.
Approach
Builds a lab-specific Isaac Sim-based simulator with a chemical reaction engine (curated substance database plus LLM-based reaction inference), a procedural lab-scene generator combining grid sampling with constraint-aware DFS, and a five-level benchmark with automated expert trajectory collection.
Key results
- ACT outperforms Diffusion Policy on most Level-1 and Level-2 tasks (e.g., Heater Beaker 86.7% vs 25.0%; Operate Drawer 73.3% vs 10.0%).
- On Level-3 generalization, pi0, ACT and DP show large ID/OOD gaps; DP drops to 8.3% OOD on Heater Beaker, and models trained on mixed object sizes reach near-zero OOD success for pick and pour.
- On Level-4 long-horizon tasks, success rates decline sharply in later sub-steps (e.g., ACT on Clean Beaker falls to 1.6% at A7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ACT rather than DP | imitation learning policies · Level-2 Heater Beaker | success rate (%): 25.0 -> 86.7 | Helped |
| Use ACT rather than DP | imitation learning policies · Level-2 Operate Drawer | success rate (%): 10.0 -> 73.3 | Helped |
| Use ACT rather than DP | imitation learning policies · Level-3 Press OOD | success rate (%): 31.6 -> 96.7 | Helped |
| Use ACT rather than DP | imitation learning policies · Level-3 Pour Liquid OOD | success rate (%): 31.6 -> 65.0 | Helped |
| Train jointly with objects of significantly different sizes | ACT and DP · Level-3 generalization to different object sizes | success rate (%): 31.2; 1.7; 25.6; 0.0 | Hurt |
| Use ACT rather than DP | imitation learning policies · Clean Beaker long-horizon task, A7 | success rate (%): 0.0 -> 1.6 | Helped |
| Execute long-horizon task sequences rather than single-stage primitive actions | ACT and DP · Level-4 long-horizon tasks | success rate (%) | Mixed |
| Generalize to objects with different shapes | current imitation learning approaches · LabBench | success rate (%) | Hurt |
Surprising findings
- Pretrained VLA pi0 did not consistently outperform models trained from scratch on Level-3 generalization tasks.
- Joint training on objects of significantly different sizes causes substantial in-domain drops and near-zero out-of-domain success.
Limitations
- Benchmark operates only in simulation; sim-to-real gap not addressed.
- Only two robot embodiments (Fetch and Panda) are supported.
- Imitation policies generalize poorly to objects with different shapes and sizes.
Experimental setup
Datasets: LabScene assets auto-collected expert demonstration trajectories PubChem-derived chemical substance database (200 compounds)
Benchmarks: LabBench
Metrics: success rate (%)
Baselines: ACT Diffusion Policy
Builds on: Isaac Sim ACT Diffusion Policy OpenPI RMPflow
Closest papers by meaning
- SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning (NeurIPS 2026 · similarity 0.87)
- Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models (NeurIPS 2026 · similarity 0.87)
- VirtualEnv: A Platform for Embodied AI Research (AAAI 2026 · similarity 0.87)
- CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter (NeurIPS 2026 · similarity 0.86)
- Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots (ICLR 2024 · similarity 0.86)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.