Research map / Multi-agent systems and games / LLM agents

Generalizable End-to-End Tool-Use RL with Synthetic CodeGym

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agent data synthesis and behavior studies · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CodeGym, a pipeline that turns coding problems into verifiable multi-turn tool-use Gym environments and trains LLM agents with GRPO, showing out-of-distribution gains on tau-bench and other benchmarks.

Problem

LLM tool-use agents trained with SFT on static trajectories or RL on narrow tasks generalize poorly to new tools and unseen workflows. A scalable, general-purpose RL environment source for agentic capabilities is lacking.

Approach

Use coding problems as a substrate: extract atomic functions from their solutions into callable tools, wrap them as OpenAI-Gym-style POMDP environments with unit-test-based binary rewards, verify solvability with pass@10 solution generation, and train with GRPO on the filtered environments.

Key results

• Qwen2.5-32B-Instruct improves average OOD score from 42.3 to 49.6 (+7.3) after CodeGym RL training; tau-retail gains +13.0.
• Qwen2.5-7B-Instruct gains only +2.8 average OOD, indicating larger models generalize better from CodeGym.
• RL on CodeGym outperforms Oracle-SFT (41.3) and distillation from Seed-1.6-Thinking (47.9) on OOD average for Qwen2.5-32B.

Surprising findings

• Long-CoT model QwQ-32B loses 3.3 points on ZebraLogic after CodeGym training despite gains in tool-use tasks.
• Qwen2.5-7B-Instruct makes the most tool calls, driven by repetitive failure-recovery loops.

Limitations

• Trained models show a large gap between tool-call counts and oracle solutions, suggesting shortcut behavior remains.
• Synthetic environments rely on LLM-generated tools, solutions and unit tests, with pass@10 verification that may miss errors.
• Only one seed for models 32B and larger due to compute limits.

Experimental setup

Datasets: KodCode

Benchmarks: tau-airline tau-retail tau2-bench ALFWorld ZebraLogic MMLU-Pro

Metrics: accuracy

Baselines: Qwen2.5-32B-Instruct (base) Oracle-SFT Distillation from Seed-1.6-Thinking CodeGym-Full (unfiltered)

Builds on: GRPO KodCode ReAct OpenAI Gym

Closest papers by meaning

• WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment (NeurIPS 2024 · similarity 0.89)
• RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (ICML 2025 · similarity 0.89)
• AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs (NeurIPS 2026 · similarity 0.89)
• Self-Challenging Language Model Agents (NeurIPS 2025 · similarity 0.89)
• Joint Optimization of Tool Creation and Use for Large Language Model Agents (NeurIPS 2026 · similarity 0.89)
• AgentRefine: Enhancing Agent Generalization through Refinement Tuning (ICLR 2025 · similarity 0.89)