Research map / Multi-agent systems and games / LLM agents
APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes APIGen-MT, a two-phase pipeline that generates verified task blueprints with LLM committee review and then simulates multi-turn human-agent trajectories, and uses the data to train xLAM-2-fc-r models (1B–70B) that top BFCL v3 and τ-bench.
Problem
Multi-turn agent training data with realistic human-agent dynamics and verifiable ground-truth actions is scarce and expensive to collect manually.
Approach
Separate data generation into a verified task "blueprint" phase (format/execution/policy checks, LLM review committee with majority voting, reflection-based feedback loops, reverse task recombination) and a simulated human-agent interplay phase that collects only trajectories that pass state-based and output-based checks.
Key results
- xLAM-2-70b-fc-r reaches 78.19% overall on BFCL v3 (rank 1 at time of leaderboard snapshot) and 75.12% multi-turn accuracy.
- On τ-bench, xLAM-2-70b-fc-r achieves 56.2% overall success, above GPT-4o (52.9%) and Llama 3.1 70B (38.2%), below Claude 3.5 Sonnet (new) at 60.1%.
- The agentic pipeline with review committee and reflection gives a 2.5x boost in task collection success rate, reaching 70%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on APIGen-MT synthetic multi-turn data | xLAM-2-70b-fc-r · τ-bench overall | success rate: 56.2 vs 52.9 | Helped |
| Train on APIGen-MT synthetic multi-turn data | xLAM-2-70b-fc-r · BFCL v3 overall | overall accuracy: 78.19 vs 74.31 | Helped |
| Train on APIGen-MT synthetic multi-turn data | xLAM-2-8b-fc-r · BFCL v3 multi-turn | multi-turn accuracy: 69.25 vs 58.75 | Helped |
| Use Best-of-N (N=4) user simulation with self-critique instead of naive user simulation | gpt-4o test assistant · τ-Retail | success rate average: 62.8 -> 67.0 | Helped |
| Use Best-of-N (N=4) user simulation with self-critique instead of naive user simulation | xLAM-2-70b-fc-r test assistant · τ-Retail | success rate average: 67.1 -> 68.8 | Helped |
| Use Best-of-N (N=4) user simulation with self-critique instead of naive user simulation | gpt-4o test assistant · τ-Retail | success rate variance: 11.1 -> 2.6 | Helped |
| Use Best-of-N (N=4) user simulation with self-critique instead of naive user simulation | xLAM-2-70b-fc-r test assistant · τ-Retail | success rate variance: 9.7 -> 4.0 | Helped |
Surprising findings
- Smaller xLAM-2 models (8B, 32B) match or exceed larger baselines such as Llama 3.1 70B on τ-bench.
- Best-of-N user simulation raised gpt-4o's average τ-bench retail success from 62.8% to 67.0% while reducing variance from 11.1 to 2.6.
Limitations
- Failed trajectories are discarded and not used as contrastive signal.
- Multi-stage validation adds computational overhead.
- Some stochasticity in simulated human behavior remains; xLAM-2-70b requires more interaction turns than Claude.
Experimental setup
Datasets: τ-bench retail and airline APIs and policies PersonaHub APIGen-MT-5k
Benchmarks: BFCL v3 τ-bench
Metrics: overall accuracy multi-turn accuracy pass@1 pass^k success rate variance
Baselines: gpt-4o o1 Claude 3.5 Sonnet Claude 3.7 Sonnet DeepSeek V3 Llama 3.1 70B Instruct Qwen 2.5 32B Instruct watt-tool-70b ToolACE-8B
Builds on: APIGen τ-bench PersonaHub Agent-FLAN LLaMA-Factory
Closest papers by meaning
- ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interaction (ICLR 2026 · similarity 0.88)
- Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning (ICML 2026 · similarity 0.88)
- Learn-by-interact: A Data-Centric Framework For Self-Adaptive Agents in Realistic Environments (ICLR 2025 · similarity 0.88)
- GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators (NeurIPS 2026 · similarity 0.87)
- TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarios (ICML 2026 · similarity 0.86)
- Agent2World: Learning to Generate Symbolic World Models via Adaptive Multi-Agent Feedback (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.