Research map / Multi-agent systems and games / LLM agents
OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Workforce, a modular hierarchical multi-agent framework (planner, coordinator, worker nodes) that reaches 69.70% on GAIA, and OWL, which trains only a domain-agnostic planner (SFT then DPO) to lift Qwen2.5-32B-Instruct by 16.37 points.
Problem
Multi-agent LLM systems for real-world task automation are domain-specific, requiring redesign at inference time and full retraining of all agents when moved to new domains.
Approach
Decouple a domain-agnostic planner/coordinator from swappable domain-specific worker nodes at inference, and train only the planner (SFT on synthesized trajectories followed by DPO on correct vs. incorrect rollouts) so the same planner transfers across domains.
Key results
- Workforce with Claude-3.7-Sonnet reaches 69.70% average on GAIA validation, above OpenAI Deep Research (67.36%) in the paper's table.
- OWL (SFT+DPO) lifts Qwen2.5-32B-Instruct from 36.36% to 52.73% on GAIA (+16.37), above GPT-4o-mini (47.27%) and Qwen2.5-72B-Instruct (49.09%).
- Training only the planner (45.45%) outperforms training only workers (31.51%) in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Workforce instead of the Single Agent baseline with the same tools | GPT-4o-based GAIA assistant · GAIA validation | accuracy: +23.03% | Helped |
| Use Workforce instead of Role Playing with the same tools | GPT-4o-based GAIA assistant · GAIA validation | accuracy: +6.06% | Helped |
| SFT the planner on OWL trajectories (ablation) | Qwen2.5-32B-Instruct planner · GAIA validation, average | accuracy: 36.36 -> 41.21 | Helped |
| SFT the planner on OWL trajectories (ablation) | Qwen2.5-32B-Instruct planner · GAIA validation, Level 3 | accuracy: 19.23 -> 15.38 | Hurt |
| Add DPO after SFT (ablation) | Qwen2.5-32B-Instruct planner · GAIA validation, average | accuracy: 36.36 -> 52.73 | Helped |
| Add DPO after SFT (ablation) | Qwen2.5-32B-Instruct planner · GAIA validation, Level 3 | accuracy: 19.23 -> 26.92 | Helped |
| Train only the planner rather than only the workers (ablation) | Workforce · GAIA validation | accuracy: 31.51% -> 45.45% | Helped |
| Train both planner and workers rather than only the planner (ablation) | Workforce · GAIA validation | accuracy: 45.45% -> 46.68% | Helped |
| Train on filtered instead of unfiltered trajectories (ablation) | OWL planner · GAIA validation | accuracy | Helped |
Surprising findings
- SFT alone regresses on GAIA Level 3 (-3.85%), and DPO is needed to recover and exceed the base model.
- Training both planner and workers gives only marginal gains (46.68%) over planner-only training (45.45%).
Limitations
- Performance depends on availability of high-quality domain-specific tools.
- RL from real-world feedback is slow due to web search latency.
- Blocking leaked GAIA websites is needed for fair comparison; GAIA leakage concerns.
Experimental setup
Datasets: HotpotQA WikiTableQuestions Math-related problems (custom curated) Infinity-MM GAIA validation
Benchmarks: GAIA validation
Metrics: accuracy (%)
Baselines: OpenAI Deep Research h2oGPTe Agent Trase Agent Langfun Agent v2.1 Open Deep Research Role Playing Single Agent GPT-4o-mini Qwen2.5-72B-Instruct Claude-3.7-Sonnet
Builds on: CAMEL MetaGPT Direct Preference Optimization LlamaFactory GAIA
Closest papers by meaning
- MoRSE: Task-Oriented Multi-Agent System with Mixture of Role-Subtask Experts (NeurIPS 2026 · similarity 0.87)
- AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents (ICLR 2026 · similarity 0.87)
- TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.87)
- TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (NeurIPS 2025 · similarity 0.87)
- SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (NeurIPS 2026 · similarity 0.87)
- WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.