Research map / Multi-agent systems and games / Human-AI and goal-conditioned
WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds an LLM-synthesized Python world model (transition and reward programs) from environment interaction, with an optimism-under-uncertainty constraint that makes the model plannable toward goals, evaluated on Sokoban, Minigrid, and AlfWorld.
Problem
Learning world models sample-efficiently and transferably from environment interaction, with efficient exploration in sparse-reward, long-horizon settings, while avoiding the per-action LLM cost of ReAct-style agents.
Approach
The LLM synthesizes an interpretable Python program for transition and reward functions, trained with a data-fit constraint plus an optimism constraint requiring the reward model to admit a goal-reaching plan from the initial state; programs are refined via REx-based debugging and reused across environments for transfer.
Key results
- ReAct succeeds on only 15%±8% of basic Sokoban levels, while the WorldCoder agent builds a basic-competence world model after 50 actions and 400k LLM tokens (about $15).
- Deep RL takes more than 1 million steps to learn 2-box Sokoban, whereas WorldCoder learns from far fewer interactions.
- On MiniGrid-UnlockPickup, WorldCoder with the optimism objective learns the correct world model within 100 actions, while PPO fails after 3x10^8 actions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the optimism objective φ2 instead of data consistency alone (ablation) | WorldCoder · MiniGrid-UnlockPickup-v0 | actions to learn the correct world model: 3×10^8 vs no more than 100 | Helped |
| Use the optimism objective φ2 (ablation) | WorldCoder · Sokoban | learning performance | No clear effect |
| Transfer the learned world model between environments (ablation) | WorldCoder · MiniGrid curriculum | episodes of exploration to solve new environments | Helped |
| Use optimism under uncertainty for goal-directed exploration (ablation) | WorldCoder · MiniGrid without transfer | ability to solve harder environments | Helped |
| Use optimism under uncertainty to update the reward model for a new goal (ablation) | WorldCoder · MiniGrid environment 3 | zero-shot instruction following | Helped |
| Generate a new reward function from the goal without enforcing φ2 (ablation) | WorldCoder · MiniGrid environment 3 | instruction-following performance | Hurt |
| Use the LLM to reason about and predict actions directly instead of building an explicit world model | Sokoban · Basic levels | success rate: 15%±8% | Hurt |
| Use WorldCoder instead of deep RL to learn basic Sokoban dynamics | Sokoban · 2-box Sokoban | environment steps to learn: >1 million | Helped |
| Use optimism under uncertainty for task exploration (ablation) | WorldCoder · AlfWorld representative tasks | nonzero task performance | Helped |
Surprising findings
- Ablating optimism (prompting GPT-4 to generate a new reward function) struggles to follow new natural-language instructions, showing LLM reward generation is unreliable.
- On Sokoban, the optimism objective has little effect because the dense reward allows easy learning through random exploration.
Limitations
- Assumes deterministic environments and discrete symbolic states, whereas real-world perception is continuous and messy.
- Only a basic mechanism for growing and transferring knowledge; no library of reusable subroutines.
- Cannot solve very hard Sokoban levels (5+ boxes) due to planning difficulty.
Experimental setup
Datasets: Sokoban Minigrid AlfWorld
Benchmarks: Sokoban Minigrid AlfWorld
Metrics: average reward per episode sample efficiency (environment steps) LLM tokens/calls success rate
Baselines: ReAct PPO prompt4reward deep model-based RL RAP
Builds on: REx Contextual MDP Object-Oriented MDP MCTS value iteration
Closest papers by meaning
- Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search (NeurIPS 2024 · similarity 0.90)
- Generalizable End-to-End Tool-Use RL with Synthetic CodeGym (ICLR 2026 · similarity 0.89)
- PoE-World: Compositional World Modeling with Products of Programmatic Experts (NeurIPS 2025 · similarity 0.89)
- Agent2World: Learning to Generate Symbolic World Models via Adaptive Multi-Agent Feedback (NeurIPS 2026 · similarity 0.89)
- Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation (ICLR 2025 · similarity 0.89)
- WALL-E: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.