Research map / Reinforcement learning / Visual and representation RL
Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Kinetix, an open-ended space of 2D physics RL environments with a JAX-based engine Jax2D, and trains a general PPO agent on billions of randomly generated levels that zero-shot solves hand-designed holdouts and fine-tunes better than tabula rasa training.
Problem
Generalist agents for sequential decision-making lack the large-scale generalisation seen in text and image models; most online RL environments are narrow and homogeneous.
Approach
A unified, expressive space of 2D rigid-body physics tasks (circles, polygons, joints, thrusters) with a fixed reward, sampled procedurally and simulated on GPU via a dynamically specified JAX physics engine, enabling billions of steps of online pretraining of a single transformer-based PPO agent with SFL level curation.
Key results
- General agent trained on random M levels for 5B steps zero-shot solves a substantial share of unseen hand-designed holdout levels, improving throughout training across S, M and L sizes.
- On Car-Ramp, tabula rasa PPO for 1B steps fails to reach the target once, while the general agent solves it zero-shot about 5% of the time and reliably after fine-tuning.
- Jax2D reaches 824K steps per second inside an RL loop versus 24K for Box2D, a speedup above 30x in RL training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Jax2D instead of Box2D in the RL training loop | RL training · Jax2D Speed Results | steps per second: 824K vs 24K | Helped |
| use Jax2D instead of Box2D for raw engine simulation | physics engine · Jax2D Speed Results | steps per second: 9049K vs 1982K | Helped |
| train with SFL-selected levels rather than the DR distribution | Kinetix general agent · Kinetix holdout levels | solve rate | No clear effect |
| fine-tune the pretrained general agent instead of training tabula rasa | Kinetix general agent · L-size holdout levels | solve rate | Helped |
| fine-tune the pretrained general agent instead of training tabula rasa | Kinetix general agent · Thruster-Large-Obstacles | learning speed | Hurt |
| pretrain on random Kinetix levels before attempting Car-Ramp | Kinetix general agent · Car-Ramp | solve rate: around 5% | Helped |
| pretrain on random Kinetix levels before fine-tuning on Car-Ramp | Kinetix general agent · Car-Ramp | solve rate | Helped |
| train on randomly generated Kinetix levels | Kinetix general agent · Small holdout levels | solve rate | Helped |
| train on randomly generated Kinetix levels | Kinetix general agent · Medium holdout levels | solve rate | Helped |
| train on randomly generated Kinetix levels | Kinetix general agent · Large holdout levels | solve rate | Helped |
Surprising findings
- Fine-tuning a general agent can solve levels (Car-Ramp, MuJoCo-Walker-Hard) where tabula rasa PPO for 1B steps never succeeds.
- PLR and ACCEL provided no improvement over domain randomisation in preliminary experiments.
Limitations
- Zero-shot generalisation is strong for S levels but weak and non-monotonic for L levels, where random levels are more often unsolvable.
- Requires billions of online environment interactions, which is impractical for real-world applications.
- Jax2D is 2D only, so transfer to real-world robotics is not a goal.
Experimental setup
Datasets: procedurally generated Kinetix levels
Benchmarks: Kinetix 74 hand-designed holdout levels
Metrics: solve rate sample efficiency of fine-tuning goal-position correlation
Baselines: Domain Randomisation (DR) PLR ACCEL tabula rasa PPO Box2D
Builds on: PPO Box2D Brax SFL PureJaxRL ACCEL
Closest papers by meaning
- Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAX (ICLR 2024 · similarity 0.87)
- CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation (ICML 2026 · similarity 0.86)
- Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversity (NeurIPS 2024 · similarity 0.86)
- LaST-R1: Reinforcing Robotic Manipulation via Adaptive Physical Latent Reasoning (NeurIPS 2026 · similarity 0.86)
- Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining (ICLR 2025 · similarity 0.86)
- JaxMARL: Multi-Agent RL Environments and Algorithms in JAX (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.