Research map / Large language models / Agents, code and math
LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LMRL-Gym, a benchmark of 8 multi-turn language RL tasks (5 capability tests, 3 interactive dialogue) with offline datasets and LLM-based simulators, plus a research framework benchmarking BC, ILQL, MC returns, PPO and GPT-4 prompting with GPT-2 models.
Problem
Existing NLP benchmarks lack multi-turn, reward-based evaluation, so RL algorithm researchers have no standard way to compare methods for goal-directed multi-turn language interaction (dialogue, games, negotiation), since human evaluation is costly.
Approach
Provide a benchmark of 8 multi-turn language RL tasks where each task has an offline dataset generated synthetically (game engines or LLM simulators) and an interactive simulator for online/offline evaluation, along with a unified toolkit of offline and online RL baselines.
Key results
- ILQL achieves the best normalized reward on FO Maze (99.9), PO Maze (76.3), FO Text-Nav (91.8) and Wordle (97.7) among baselines.
- Online PPO attains the best result on Endgames (77.5) and Chess (48.0), and outperforms offline methods on PO Text-Nav (85.5).
- GPT-4 few-shot prompting underperforms RL fine-tuning on most tasks, scoring 0 on Chess and Endgames, but does best on 20Qs (95.7) and Guess (92.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use online PPO instead of offline value-based RL | GPT-2 policies · PO Text-Nav | normalized reward: 83.7 -> 85.5 | Helped |
| Use online PPO instead of offline value-based RL | GPT-2 policies · Chess | normalized reward: 47.3 -> 48.0 | Helped |
| Use online PPO instead of offline value-based RL | GPT-2 policies · Endgames | normalized reward: 50.2 -> 77.5 | Helped |
| Use ILQL instead of MC Returns | GPT-2 offline RL · Maze, Text-Nav, and Wordle | normalized reward | Helped |
| Use ILQL instead of MC Returns | GPT-2 offline RL · Endgames, 20Qs, Guess My City, and Car Dealer | normalized reward | Hurt |
| Use online PPO instead of Online Filtered BC | GPT-2 policies · All tasks except Wordle | normalized reward | Helped |
| Use GPT-4 few-shot prompting instead of RL fine-tuning | GPT-4 · Most tasks | normalized reward | Hurt |
| Evaluate in a partially observed rather than fully observed setting | All algorithms · Maze and Text-Nav | normalized reward | Hurt |
Surprising findings
- MC Returns outperforms ILQL on the complex-language tasks (Endgames, 20Qs, Guess, Car Dealer), reversing the ordering seen on symbolic tasks.
- GPT-4 few-shot prompting scores 0 normalized reward on Chess and Endgames, far below fine-tuned GPT-2 policies.
Limitations
- Relies on smaller GPT-based LLMs for data generation and simulators, so simulators may deviate from human behavior.
- Models trained and evaluated at most 1.5B parameters.
- PPO shows training instabilities that hyperparameter tuning did not resolve.
Experimental setup
Datasets: LMRL-Gym Maze LMRL-Gym Text-Nav LMRL-Gym Wordle LMRL-Gym Chess LMRL-Gym Endgames LMRL-Gym 20Qs LMRL-Gym Guess My City LMRL-Gym Car Dealer
Benchmarks: LMRL-Gym
Metrics: normalized reward raw average reward success rate
Baselines: Behavior cloning Filtered BC Monte-Carlo returns ILQL Online PPO Online Filtered BC GPT-4 few-shot prompting
Builds on: ILQL PPO GPT-2 TextWorld
Closest papers by meaning
- ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL (ICML 2024 · similarity 0.91)
- Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.90)
- Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF (ICLR 2025 · similarity 0.90)
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.90)
- ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.90)
- From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.