Research map / Multi-agent systems and games / LLM agents
Using Unity to Help Solve Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces United Unity Universe (U3), an open-source Unity-based toolkit for procedurally generating XLand-style 3D RL environments, plus OpenXLand datasets and a SAC baseline.
Problem
Open-source environments for open-ended, procedurally generated 3D RL and meta-RL are limited in scope and controllability, while XLand and XLand 2.0 are not open-source.
Approach
A Unity-based ecosystem with wave-function-collapse world generation and an extensible production-rule system (conjunctive conditions, extra actions and objects) that reimplements XLand 2.0 as OpenXLand, with two-way Unity-Python communication for dynamic curricula.
Key results
- SAC trained for 75M steps on a single easy task reaches higher episodic return than a random agent, but shows minimal improvement over random on the full easy dataset.
- Throughput increases with the number of parallel Unity instances before saturating; larger CPU core counts raise saturation point.
- Six million pre-generated worlds and production-rule instances are released, claimed to span 36 trillion unique environment combinations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train SAC on a single easy task for 75 million environment steps | SAC in OpenXLand · Single easy environment | episodic return | Helped |
| Train SAC across tasks, changing the task each episode | SAC in OpenXLand · Easy dataset | episodic return | No clear effect |
| Train SAC across tasks, changing the task each episode | SAC in OpenXLand · Medium dataset | reward collection | Hurt |
| Increase environment size from 10x105 to 200x10x5 | U3 · Unity environment performance scaling | script time (ms): 3.04 -> 6.77 | Hurt |
| Increase production objects from 3 to 300 | U3 · 10x10x5 environment, 2 production rules | script time (ms): 3.04 -> 65.29 | Hurt |
| Increase production rules from 2 to 2000 | U3 · 10x10x5 environment, 3 production objects | script time (ms): 3.04 -> 10.44 | Hurt |
| Increase world difficulty from Easy to Hard | OpenXLand · Environment generation | average environment generation time (ms): 0.6 -> 172.8 | Hurt |
Surprising findings
- SAC shows minimal improvement over a random agent when trained on the full easy dataset, and random agents earn no reward on medium datasets.
Limitations
- Installing and running Unity is nontrivial and hampers reproducibility; a Docker environment is planned.
- Missing features such as the 'see' condition, passing active rules to the agent, and teacher models/auto-curricula.
- Compute cost of training agents in Unity is high relative to gridworld alternatives.
Experimental setup
Datasets: OpenXLand procedurally generated worlds OpenXLand production rules
Benchmarks: OpenXLand easy_low / short_few OpenXLand medium datasets
Metrics: episodic return 20th percentile normalized return steps per second (SPS)
Baselines: Soft Actor-Critic random agent
Builds on: XLand XLand 2.0 Wave Function Collapse Unity ML-Agents MemoryMaze
Closest papers by meaning
- XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX (NeurIPS 2024 · similarity 0.85)
- XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning (ICLR 2025 · similarity 0.85)
- Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAX (ICLR 2024 · similarity 0.85)
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.84)
- Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning (ICML 2024 · similarity 0.84)
- Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.