Research map / Reinforcement learning / Visual and representation RL

Democratizing Tool Learning with Environments Fully Simulated by a Free 8B Language Model

NeurIPS 2026 · Reinforcement learning › Visual and representation RL · approach: Scalable RL systems and environments · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TRUSTEE, which trains tool-calling agents by RL in fully simulated environments (task generation, user, tool, and verifier simulation) backboned by a free Qwen3-8B, plus adaptive curriculum over task difficulty.

Problem

RL for tool-calling agents needs online environments, but existing approaches rely on annotated data or costly proprietary LMs to build environments that stay fixed once created.

Approach

An open-source LM generates tasks, simulates users and tools, and judges trajectories with a 3-level reward, while difficulty (tool count, turns, system prompt specificity, user persona/ambiguity, rewarding criteria) adapts online based on batch reward thresholds.

Key results

• Qwen3-8B-Base with TRUSTEE reaches 7.88 on BFCL Multi-Turn vs 0.00 base and 0.62 for EnvScaler; Non-Live 83.92 vs 75.29 for EnvScaler.
• Qwen3-8B with TRUSTEE scores 46.38 on BFCL Multi-Turn vs 44.50 for EnvScaler and 39.88 base, and best on 5 of 6 subsets.
• Removing adaptive curriculum drops Qwen3-8B-Base Multi-Turn from 7.88 to 3.38.

Surprising findings

• Qwen3-8B serving as its own environment backbone yields the best results on Qwen3-8B, and a stronger Qwen3.5-397B backbone did not consistently improve results.
• Simia-RL checkpoint severely degrades on most BFCL subsets (Multi-Turn 2.00 vs 39.88 base) despite strong Airline performance.

Limitations

• Scalability limited by the bounded capability of the 8B simulator for many tools and many turns, which may yield inaccurate rewards.
• Many hyper-parameters in adaptive curriculum make tuning difficult.
• Evaluated only on stateless REST-style tools, not stateful tools.

Experimental setup

Datasets: ToolBench tools (RapidAPI REST APIs)

Benchmarks: BFCL V4 Non-Live BFCL V4 Live BFCL V4 Multi-Turn tau2-bench Airline tau2-bench Retail tau2-bench Telecom

Metrics: accuracy

Baselines: ToolRL Tool-N1 EnvScaler AWM Simia-RL

Builds on: GRPO ToolBench verl vLLM YaRN

Closest papers by meaning

• Joint Optimization of Tool Creation and Use for Large Language Model Agents (NeurIPS 2026 · similarity 0.89)
• ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
• ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Training (ICML 2026 · similarity 0.88)
• Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning (ICLR 2026 · similarity 0.87)
• ToolRL: Reward is All Tool Learning Needs (NeurIPS 2025 · similarity 0.87)
• Generalizable End-to-End Tool-Use RL with Synthetic CodeGym (ICLR 2026 · similarity 0.87)