Research map / Reinforcement learning / Policy gradient and actor-critic

Monte Carlo Planning with Large Language Model for Text-Based Game Agents

ICLR 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MC-DML, which uses GPT-3.5 as a PUCT prior with in-trial and reflection-based cross-trial memory to plan in Jericho text games without iterative planning-then-learning.

Problem

Planning in text-based adventure games with sparse rewards and bottleneck states is slow for MCTS+RL methods that need multiple iterations, and MCTS lacks language understanding for action exploration.

Approach

Use an LLM as the non-uniform prior policy inside PUCT, conditioned on in-trial trajectory memory and cross-trial reflections generated from failed trajectories, so action value estimates adapt during a single planning session.

Key results

• MC-DML reaches 48.66 on Zork1 at initial planning, versus 45.2 for MC-LAVE-RL after 4 iterations.
• On Deephome MC-DML scores 67 versus 35 for MC-LAVE-RL.
• MC-DML completes Pentari (70) and reaches 346.67 on Detective, and matches or beats baselines in 8 of 9 games against LLM/MCTS baselines.

Measured effects reported in the paper

Surprising findings

• A plain LLM agent scores near zero on most games, suggesting it lacks walkthrough knowledge under the prompting setup.
• Disabling dynamic pruning has little effect on most games, though it substantially helps Ztuu.

Limitations

• In-trial memory is a short trailing window, so clues from far earlier in the game are not retained.
• Relies on token log-probabilities from GPT-3.5; APIs without them need self-consistency or verbalized estimates not evaluated here.
• Does not match RL baselines on some games such as Ludicorp, Library and Balances in the table where it is lower or tied.

Experimental setup

Datasets: Jericho benchmark games (Zork1, Deephome, Ludicorp, Pentari, Detective, Library, Balances, Temple, Ztuu)

Benchmarks: Jericho

Metrics: game score

Baselines: DRRN KG-A2C-Hard MC!Q*BERT MPRC-DQN RC-DQN BiKE+CBR LLM agent Reflection agent PUCT-RL MC-LAVE-RL

Builds on: PUCT MC-LAVE-RL LLM-MCTS Reflexion

Closest papers by meaning

• Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Search (ICLR 2025 · similarity 0.89)
• Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search (NeurIPS 2024 · similarity 0.89)
• Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.89)
• On the Modeling Capabilities of Large Language Models for Sequential Decision Making (ICLR 2025 · similarity 0.89)
• Mastering Board Games by External and Internal Planning with Language Models (ICML 2025 · similarity 0.88)
• $\texttt{Multi}^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments (ICML 2026 · similarity 0.88)