Research map / Reinforcement learning / Visual and representation RL
Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Craftax-Classic (a JAX rewrite of Crafter running ~250x faster) and Craftax (a much harder NetHack-inspired extension), and benchmarks PPO, RNN, exploration bonuses and UED methods on them.
Problem
Existing open-ended RL benchmarks are either too slow to run without large compute (Crafter, NetHack, Minecraft) or too simple to pose a significant challenge (Minigrid, Procgen).
Approach
Reimplement Crafter fully in JAX so the whole RL pipeline runs on a GPU, then extend it with NetHack/roguelike mechanics (9 floors, 19 creatures, potions, enchantments, 65 achievements) to create a hard but fast open-ended benchmark.
Key results
- Craftax-Classic runs up to ~250x faster than Crafter (405,618 vs 1,580 steps/s best case), and PPO reaches ~90% of optimal reward with 1B steps in under an hour on one GPU.
- On Craftax-1B, PPO and PPO-RNN master basic achievements but no tested method makes progress in the hardest achievement categories; E3B significantly reduces reward.
- In UED experiments on Craftax-1B, PLR performs best, while Robust PLR only slightly outperforms domain randomisation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add recurrent memory with a GRU to PPO | PPO on Craftax-1B · Craftax-1B | reward | Helped |
| Add intrinsic exploration rewards using ICM, RND, or E3B | PPO on Craftax-1B · Craftax-1B | reward | No clear effect |
| Add E3B episodic exploration bonuses | PPO on Craftax-1B · Craftax-1B | reward | Hurt |
| Train for 10 billion rather than 1 billion timesteps | PPO-RNN · Craftax-1B | reward | No clear effect |
| Use recurrent memory with a GRU rather than a feedforward MLP | PPO on Craftax-1M · Craftax-1M | reward | No clear effect |
| Use vanilla PLR rather than Domain Randomisation | UED methods on Craftax-1B · Craftax-1B | evaluation reward | Helped |
| Use Robust PLR rather than vanilla PLR (ablation) | PLR on Craftax-1B · Craftax-1B | evaluation reward | Hurt |
| Use unrestricted tile-swap mutations in ACCEL rather than other UED methods (ablation) | ACCEL on Craftax-1B · Craftax-1B | evaluation reward | Hurt |
| Use restricted tile-swap mutations in ACCEL rather than Domain Randomisation (ablation) | ACCEL on Craftax-1B · Craftax-1B | evaluation reward | No clear effect |
| Use noise mutations in ACCEL rather than Domain Randomisation (ablation) | ACCEL on Craftax-1B · Craftax-1B | evaluation reward | No clear effect |
| Use optimistic environment resets instead of generating a reset world for every worker (ablation) | Craftax JAX training pipeline · Craftax | speedup: around 2x | Helped |
| Use symbolic rather than pixel-based observations | Craftax · Craftax | runtime: around 10x | Helped |
Surprising findings
- Intrinsic exploration bonuses (ICM, E3B, RND) did not improve performance over plain PPO on Craftax-1B, and E3B significantly reduced reward.
- Running PPO for 10 billion timesteps barely improves performance over 1 billion, suggesting the plateau is an artefact of learning rate decay.
Limitations
- Existing methods make little progress on Craftax, so the benchmark's difficulty may be prohibitive for current algorithms.
- Optimistic environment resets introduce slight duplication of worlds, and creature counts are capped due to JAX fixed-shape constraints.
Experimental setup
Datasets: Craftax-Classic Craftax
Benchmarks: Craftax-1B Craftax-1M Craftax-Classic
Metrics: episode return as percentage of maximum (226) achievement success rate steps per second
Baselines: PPO PPO-RNN RND ICM E3B PLR Robust PLR ACCEL Domain Randomisation
Builds on: PureJaxRL Crafter Gymnax PLR ACCEL RND
Closest papers by meaning
- SPRING: Studying Papers and Reasoning to play Games (NeurIPS 2023 · similarity 0.88)
- Craftium: Bridging Flexibility and Efficiency for Rich 3D Single- and Multi-Agent Environments (ICML 2025 · similarity 0.88)
- XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAX (ICLR 2024 · similarity 0.87)
- Position: Benchmarking is Limited in Reinforcement Learning Research (ICML 2024 · similarity 0.87)
- RL-GPT: Integrating Reinforcement Learning and Code-as-policy (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.