Research map / Reinforcement learning / Visual and representation RL
RL-GPT: Integrating Reinforcement Learning and Code-as-policy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
RL-GPT uses a slow GPT-4 agent to decide which Minecraft sub-actions to code and a fast GPT-4 agent to write code and RL configurations, with a critic agent driving two-loop iteration, so that coded actions can be inserted into the RL action space.
Problem
LLM agents are strong at coding high-level plans but weak at precise low-level control in embodied open-world tasks such as Minecraft, while RL learns low-level control but is sample-inefficient on long-horizon tasks.
Approach
Treat the RL training pipeline as a tool: a slow LLM agent decides which sub-actions are codeable versus to be learned with RL, a fast LLM agent writes code and RL configs (including inserting coded high-level actions into the RL action space), and a critic agent drives iterative refinement of both agents.
Key results
- On selected MineDojo tasks RL-GPT reaches 0.65 success on crafting a table and 0.67 on a furnace, beating Plan4MC (0.30 and 0.37) and MineAgent (near 0).
- On ObtainDiamond RL-GPT achieves about 8% success with roughly 3M samples, versus 20% for VPT (16.8B samples) and 2% for DreamerV3 (100M samples).
- Ablation shows a single agent gets 0.34 on crafting a table, slow+fast 0.52, and slow+fast+critic 0.65.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the critic agent to the slow and fast agents (ablation) | RL-GPT · MineDojo: crafting table | success rate: 0.52 -> 0.65 | Helped |
| add the critic agent to the slow and fast agents (ablation) | RL-GPT · MineDojo: stone pickaxe | success rate: 0.56 -> 0.67 | Helped |
| split tasks between slow, fast, and critic agents instead of one agent (ablation) | RL-GPT · MineDojo: crafting table | success rate: 0.34 -> 0.65 | Helped |
| split tasks between slow, fast, and critic agents instead of one agent (ablation) | RL-GPT · MineDojo: stone pickaxe | success rate: 0.42 -> 0.67 | Helped |
| iterate the two-loop optimization to Iter-2 instead of zero-shot (ablation) | RL-GPT · MineDojo: crafting table | success rate: 0.26 -> 0.56 | Helped |
| iterate the two-loop optimization to Iter-2 instead of zero-shot (ablation) | RL-GPT · MineDojo: milk bucket | success rate: 0.79 -> 0.88 | Helped |
| iterate the two-loop optimization to Iter-2 instead of zero-shot (ablation) | RL-GPT · MineDojo: stone pickaxe | success rate: 0.53 -> 0.67 | Helped |
| iterate the two-loop optimization to Iter-2 instead of zero-shot (ablation) | RL-GPT · MineDojo: bed | success rate: 0.32 -> 0.30 | Hurt |
| use Iter-3 instead of Iter-2 (ablation) | RL-GPT · MineDojo: crafting table | success rate: 0.56 -> 0.65 | Helped |
| use Iter-3 instead of Iter-2 (ablation) | RL-GPT · MineDojo: milk bucket | success rate: 0.88 -> 0.93 | Helped |
| use Iter-3 instead of Iter-2 (ablation) | RL-GPT · MineDojo: stone pickaxe | success rate: 0.67 -> 0.67 | No clear effect |
| use Iter-3 instead of Iter-2 (ablation) | RL-GPT · MineDojo: bed | success rate: 0.30 -> 0.32 | Helped |
Surprising findings
- Naive PPO with sparse rewards achieves about 0% on all selected MineDojo tasks, and DEPS-BC reaches only 0.6% on ObtainDiamond.
- Inserting coded high-level actions into the RL action space outperformed using LLM-designed reward functions (0.585 vs 0.418 success rate).
Limitations
- ObtainDiamond success (about 8%) is far below DEPS-Oracle (about 60%) and relies on human-written code examples and a GPT-4 task planner.
- Evaluated only on a selected subset of MineDojo tasks in a single game environment.
- Dependence on proprietary GPT-4 API for slow, fast, and critic agents.
Experimental setup
Datasets: MineDojo
Benchmarks: MineDojo ObtainDiamond
Metrics: success rate samples dead loop ratio
Baselines: MineAgent MineAgent (AutoCraft) Plan4MC DreamerV3 VPT DEPS-BC DEPS-Oracle Pure RL Pure Code
Builds on: Code-as-Policies Plan4MC DEPS MineAgent MineCLIP PPO
Closest papers by meaning
- Generalizable End-to-End Tool-Use RL with Synthetic CodeGym (ICLR 2026 · similarity 0.89)
- AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs (NeurIPS 2026 · similarity 0.88)
- AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning (NeurIPS 2025 · similarity 0.87)
- Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning (ICLR 2026 · similarity 0.87)
- JaxMARL: Multi-Agent RL Environments and Algorithms in JAX (NeurIPS 2024 · similarity 0.87)
- Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.