Research map / Large language models / Agents, code and math
Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GIF-MCTS, an MCTS-based LLM code generation method that synthesizes Python code world models for model-based RL, evaluated on APPS Competition, a new 18-environment CWMB, and RTFM.
Problem
Generating accurate, executable Python world models (Code World Models) for model-based RL with LLMs, which requires following complex instructions, writing exact non-trivial logic, and self-debugging long programs using unit tests and trajectories.
Approach
Frame MCTS nodes as multi-line code blocks (state plus rollout) and use three code-specific actions (generate, improve, fix) with test-based feedback, plus a modified UCT that penalizes repeated same-type actions and a learned value estimate for unvisited nodes.
Key results
- On APPS Competition, GIF-MCTS with Llama 3 70B reaches 28.3% strict accuracy (pass@20) vs 25.1% for WorldCoder re-implementation and 23.2% for zero-shot CoT.
- On CWMB discrete environments with GPT-4 Turbo (budget 10), GIF-MCTS gets 0.91 accuracy and 0.81 normalized return vs 0.87 and 0.79 for WorldCoder.
- On RTFM with GPT-4 Turbo at budget 50, GIF-MCTS reaches accuracy and normalized return of 1.00 vs 0.64 accuracy for WorldCoder.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GIF-MCTS instead of WorldCoder | Llama 3 70B code world model generation · CWMB discrete action space | Accuracy: 0.79$\pm$0.04 -> 0.84$\pm$0.03 | Helped |
| Use GIF-MCTS instead of WorldCoder | Llama 3 70B code world model generation · CWMB continuous action space | Normalized return: 0.19$\pm$0.01 -> 0.22$\pm$0.01 | Helped |
| Use GIF-MCTS instead of WorldCoder | GPT-4 Turbo code world model generation · CWMB continuous action space | Accuracy: 0.24$\pm$0.06 -> 0.40$\pm$0.03 | Helped |
| Use GIF-MCTS instead of WorldCoder | GPT-4 Turbo code world model generation · RTFM simplified, budget 50 | Normalized return: -0.06 $\pm$ 0.12 -> 1.00 $\pm$ 0.00 | Helped |
| Add Generate action (ablation) | GIF-MCTS · CWMB continuous action space | Normalized return: 0.16$\pm$0.01 -> 0.23$\pm$0.02 | Helped |
| Add Generate action (ablation) | GIF-MCTS · CWMB discrete action space | Accuracy: 0.87$\pm$0.07 -> 0.88$\pm$0.07 | No clear effect |
| Add Improve action (ablation) | GIF-MCTS · CWMB continuous action space | Normalized return: 0.17$\pm$0.02 -> 0.23$\pm$0.02 | Helped |
| Add Fix action (ablation) | GIF-MCTS · CWMB discrete action space | Normalized return: 0.55$\pm$0.05 -> 0.83$\pm$0.06 | Helped |
| Add Fix action (ablation) | GIF-MCTS · CWMB continuous action space | Accuracy: 0.21$\pm$0.08 -> 0.38$\pm$0.04 | Helped |
Surprising findings
- Direct LLM world-model predictions via GPT-4 Turbo were never accurate, while the generated code step function was 4-7 orders of magnitude faster.
- Ablation shows the fix action is the most important action, and improve the least impactful.
Limitations
- Assumes deterministic, fully observable environments with natural language descriptions detailed enough to write code.
- Relies on available test cases/trajectories to evaluate code; self-generated tests left for future work.
- CWMs struggle on complex physics-based environments such as Humanoid; Llama 3 failed to produce executable CWMs for the two hardest ones.
Experimental setup
Datasets: APPS (Competition split) Code World Models Benchmark (CWMB) RTFM (modified)
Benchmarks: APPS Competition CWMB RTFM
Metrics: strict accuracy CWM transition accuracy normalized return
Baselines: Zero-shot CoT WorldCoder CodeRL Parsel CQL
Builds on: WorldCoder LATS Chain-of-Thought UCT CodeRL
Closest papers by meaning
- WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment (NeurIPS 2024 · similarity 0.90)
- RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (ICML 2025 · similarity 0.89)
- Generalizable End-to-End Tool-Use RL with Synthetic CodeGym (ICLR 2026 · similarity 0.89)
- Monte Carlo Planning with Large Language Model for Text-Based Game Agents (ICLR 2025 · similarity 0.89)
- Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.88)
- Steering Large Language Models between Code Execution and Textual Reasoning (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.