Research map / Multi-agent systems and games / LLM agents

Synthesis Through Simulation: Generating Coherent Enterprise Data via Scalable Agent-System Interaction

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Synthesis Through Simulation (STS), generating enterprise database snapshots by having an LLM agent (Generalist Populator) issue API calls against policy-enforcing mock environments, guaranteeing structural validity without schema access.

Problem

Generating realistic, structurally valid enterprise database snapshots for training and evaluating tool-calling agents, when schemas and production data are restricted and statistical synthesizers lack seed data and cannot enforce API-level business rules.

Approach

Generate data by executing operations through the policy-enforcing tool API rather than writing rows directly, so validity holds by construction and the agent only needs to address distributional fidelity; the Generalist Populator uses explore/plan/execute phases with a persistent exploration manifest and needs only the raw tool registry.

Key results

• GP achieves VPR = 1.0 across all ten environments, with average MDF of 0.88-ish and 100% constraint satisfaction without DB schema access.
• On airline, EnvScaler fails with 82% zero-write trajectories while GP wins on all distributional metrics.
• SFT on STS-populated airline environment raises pass@k from 0.286 (base) to 0.500, versus 0.375 for SFT on seed snapshot.

Surprising findings

• Schema-privileged EnvScaler fails 82% of airline trajectories due to stale entity IDs, while the schema-free GP succeeds.
• Removing the plan phase lowers all fidelity metrics even though the agent writes far more data per trajectory.

Limitations

• Validity guarantee depends on the API enforcement layer being complete with respect to business rules.
• Fidelity is sensitive to the underlying LLM and the two-tier distribution mechanism introduces model-specific biases.
• CDF is low in Forum and Banking due to task-ratio miscalibration; downstream training evaluated only on one small SFT setup.

Experimental setup

Datasets: tau2-bench airline seed snapshot

Benchmarks: tau2-bench airline STS ten enterprise environments

Metrics: VPR MDF PDF CDF WO pass@k AUC PPT progress

Baselines: CTGAN TVAE SDV HMA REaLTabFormer EnvScaler

Builds on: CTGAN TVAE SDV HMA REaLTabFormer EnvScaler tau-bench LangGraph

Closest papers by meaning

• EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings (ICML 2026 · similarity 0.87)
• Towards Direct Latent-Space Synthesis for Parallel Branches in LLM-Agent Workflows (NeurIPS 2026 · similarity 0.86)
• DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle (ICLR 2026 · similarity 0.85)
• Do Enterprise Systems Need Learned World Models? The Importance of Context to Infer Dynamics (NeurIPS 2026 · similarity 0.84)
• Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows? (NeurIPS 2024 · similarity 0.84)
• SciAgentGym: Benchmarking Multi-Step Scientific Tool-Use in LLM Agents (ICML 2026 · similarity 0.84)