Research map / Multi-agent systems and games / LLM agents

AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AgentSynth, a cost-efficient pipeline that synthesizes over 6,000 computer-use tasks and trajectories in OSWorld by chaining LLM-generated subtasks and summarizing them into difficulty-graded long-horizon tasks.

Problem

Human-annotated computer-use agent datasets are expensive and unscalable, and existing synthetic pipelines produce either unreliable trajectories for complex tasks or low-diversity, simple tasks.

Approach

Exploit information asymmetry: generate simple subtasks step-by-step in the forward direction (easy to execute and verify), then summarize the chain into a long-horizon task whose difficulty is set by the number of subtasks, so generation is easy while evaluation is hard.

Key results

• o4-mini success drops from 18% at difficulty level 1 to 4% at levels 5 and 6 in bare-agent evaluation; humans reach 70% on level 6.
• Pipeline costs $0.60 per trajectory; human-verified quality metrics (feasibility, coherence, persona relevance, verifier accuracy) all at 88% or above.
• Direct-instruction generation success falls from 64% (Easy) to 11% (Hard), while AgentSynth keeps generation success at 52-65% across levels; Agent S3 with GPT-5.1 drops from 62% at level 1 to 14% at level 6.

Surprising findings

• Stronger generator models (GPT-5, GPT-5.1) produced slightly lower evaluation success rates, suggesting they yield harder or more diverse tasks.
• Direct-instruction tasks that were solvable remained relatively easy for the evaluator, with evaluation success flat across difficulty.

Limitations

• Task generation and execution rely on proprietary GPT models, so results depend on API access and may shift with model versions.
• Verifier accuracy is validated on a small sample (100 instances, kappa 0.74), with a 12% false-accept rate on near-miss perturbations.
• Evaluation uses bare LLM agents with minimal scaffolding as a lower bound; no fine-tuning on AgentSynth trajectories is reported.

Experimental setup

Datasets: OSWorld

Benchmarks: AgentSynth

Metrics: success rate generation success rate cost per trajectory

Baselines: OS-Genesis Learn-by-interact Evol-Instruct AgentTrek Explorer InSTA Agent S2.5 Agent S3

Builds on: OSWorld ReAct WebJudge Persona Hub Agent S3

Closest papers by meaning

• AOrchestra: Automating Sub-Agent Creation for Agentic Orchestration (ICML 2026 · similarity 0.89)
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (NeurIPS 2026 · similarity 0.88)
• AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials (ICLR 2025 · similarity 0.88)
• TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.88)
• Agent S: An Open Agentic Framework that Uses Computers Like a Human (ICLR 2025 · similarity 0.88)
• Scaling Synthetic Task Generation for Agents via Exploration (ICLR 2026 · similarity 0.88)