Research map / Multi-agent systems and games / LLM agents

Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning

ICML 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent collaboration and safety · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Agent World Model (AWM), a pipeline that synthesizes 1,000 code-driven, SQLite-backed MCP tool-use environments and uses them for GRPO-based multi-turn RL of Qwen3 agents with code-augmented LLM-as-a-Judge rewards.

Problem

Agentic RL for tool-use LLMs is limited by the scarcity of diverse, reliable, executable environments; LLM-simulated environments hallucinate state transitions and are costly per step.

Approach

Synthesize full executable environments from scenario names via staged LLM generation (scenario, tasks, SQLite database and sample data, toolset schema, MCP code, verification code) with execution-based self-correction, and train agents with rewards from a code-augmented LLM judge that sees database state diffs.

Key results

• On BFCLv3, the 8B agent improves overall score from 53.83 (base) to 65.94 with AWM training, versus 52.53 for Simulator and 36.83 for EnvScaler.
• On MCP-Universe, the 8B agent reaches 11.17 overall vs 6.70 base; on tau^2-bench 14B Pass@1 rises from 36.69 to 39.03.
• The pipeline reaches over 85% first-attempt success, with about 1.13 self-correction iterations on average; 74% of environments contain at least one bug.

Surprising findings

• EnvScaler regresses on BFCLv3 (-8.93 average) and MCP-Universe (-1.39), while AWM improves over base on all benchmarks.
• Code-only verification underperforms code-augmented judging and can produce false negatives from environment imperfections.

Limitations

• Training used only 526 of the 1,000 environments due to limited compute, and mainly Qwen3 4B/8B/14B models.
• Synthetic environments are imperfect: 74% contain at least one bug and about 4% of RL rollouts hit runtime errors.
• Fixed generation process with no self-evolving loop; robustness and safety to adversarial inputs and access control not evaluated.

Experimental setup

Datasets: AWM synthetic environments (1,000 scenarios, 10,000 tasks) tau^2-bench BFCLv3 MCP-Universe

Benchmarks: BFCLv3 tau^2-bench MCP-Universe

Metrics: overall accuracy Pass@1 Pass@4 task success rate Fleiss' kappa pairwise agreement

Baselines: Base (Qwen3) Simulator (GPT-5 simulated environments) EnvScaler tau^2-bench LLM-only verification code-only verification

Builds on: GRPO verl AgentFly Self-Instruct Model Context Protocol

Closest papers by meaning

• Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.90)
• GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators (NeurIPS 2026 · similarity 0.89)
• Test-Time Adaptation for LLM Agents via Environment Interaction (ICLR 2026 · similarity 0.88)
• $\texttt{Multi}^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments (ICML 2026 · similarity 0.88)
• Generalizable End-to-End Tool-Use RL with Synthetic CodeGym (ICLR 2026 · similarity 0.88)
• APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay (NeurIPS 2025 · similarity 0.88)