Research map / Multi-agent systems and games / LLM agents
OpenHands: An Open Platform for AI Software Developers as Generalist Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OpenHands, an MIT-licensed open platform for building and evaluating generalist AI software agents with sandboxed runtime, event-stream architecture, agent hub, multi-agent delegation, and evaluation on 15 benchmarks.
Problem
Building, evaluating, and safely deploying AI agents that interact with the world through software (code, shell, web) lacks a unified, reusable, open platform spanning agents, sandboxed environments, and benchmarks.
Approach
A community-driven platform combining an event-stream agent abstraction, Docker-sandboxed runtime (bash, IPython, browser), a CodeAct-based generalist agent with delegation to specialist agents, an AgentSkills library, and an integrated 15-benchmark evaluation framework with LLM-mocked integration tests.
Key results
- CodeActAgent v1.8 with claude-3-5-sonnet resolves 26.0% of SWE-Bench Lite instances without hints.
- CodeActAgent v1.5 with gpt-4o reaches 79.3% on HumanEvalFix Python, versus 48.6% for StarCoder2-15B prompted.
- OpenHands CodeActAgent with gpt-4o achieves 53.1% on GPQA diamond, above few-shot GPT-4 CoT at 38.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CodeAct general action space (bash, Python, browser) with the same system prompt across categories | CodeActAgent v1.8 · SWE-Bench Lite / WebArena / GPQA / GAIA | Success Rate (%) | No clear effect |
| Delegate web browsing subtasks from CodeActAgent to specialized BrowsingAgent (ablation) | CodeActAgent v1.8 with gpt-4o-2024-05-13 · WebArena | Success Rate (%): 14.5 vs 14.8 for BrowsingAgent alone | No clear effect |
| Delegate web browsing subtasks from CodeActAgent to specialized BrowsingAgent (ablation) | CodeActAgent v1.8 with gpt-4o-2024-05-13 · MiniWoB++ | Success Rate (%): 39.8 vs 40.8 for BrowsingAgent alone | Hurt |
| Upgrade base model from gpt-4o-mini to gpt-4o in CodeActAgent v1.8 | CodeActAgent v1.8 · SWE-Bench Lite | Success Rate (%): 7.0 to 22.0 | Helped |
| Use Claude-3.5-Sonnet instead of GPT-4o as backbone for CodeActAgent v1.8 | CodeActAgent v1.8 · SWE-Bench Lite | Success Rate (%): 22.0 to 26.0 at lower average cost (1.10 vs 1.72) | Helped |
| Use CodeActAgent with strong model vs weaker model (gpt-3.5 to gpt-4o) (ablation) | CodeActAgent v1.5 · ToolQA easy | Success Rate (%): 2.3 to 47.2 | Helped |
| Use gpt-4o-mini vs gpt-4o-based BrowsingAgent (ablation) | BrowsingAgent v1.0 · WebArena | Success Rate (%): 8.5 to 14.8 | Helped |
| Replace gpt-3.5-turbo with gpt-4o in CodeActAgent for AgentBench OS (ablation) | CodeActAgent v1.5 · AgentBench OS (bash) | Success Rate (%): 11.8 to 57.6 | Helped |
| Add CodeAct-based multi-turn tool use and code execution (CodeActAgent v1.5 with gpt-4o) on HumanEvalFix | CodeActAgent v1.5 · HumanEvalFix Python | Success Rate (%): 79.3 vs best non-agentic 48.6 (StarCoder2-15B) | Helped |
| Add search/web-browsing tool capability to OpenHands on GPQA diamond (gpt-4o vs prior CoT) | OpenHands CodeActAgent v1.5 with GPT4o · GPQA diamond | Success Rate (%): 53.1 vs 38.8 for few-shot CoT GPT-4 | Helped |
Surprising findings
- Performance on the BIRD and GPQA benchmarks is competitive without benchmark-specific prompt engineering, with the same CodeAct agent prompt used across categories.
- Gpt-3.5-turbo-0125 scores only 2.3% on ToolQA under CodeActAgent v1.5, far below gpt-4o at 47.2%.
Limitations
- Agents still struggle with complex tasks and long-file editing.
- Workflow construction still requires substantial handcrafted effort.
- Results rely on closed API models and are not trained or optimized by the platform itself.
Experimental setup
Datasets: SWE-Bench SWE-bench Lite HumanEvalFix BIRD BioCoder ML-Bench Gorilla APIBench ToolQA WebArena MiniWoB++ GAIA GPQA
Benchmarks: SWE-Bench Lite HumanEvalFix BIRD ML-Bench BioCoder Gorilla APIBench ToolQA WebArena MiniWoB++ GAIA GPQA AgentBench
Metrics: success rate resolve rate execution accuracy pass@k average cost per instance
Baselines: SWE-Agent AutoCodeRover Aider Moatless Tools Agentless WebArena Agent AutoGPT GPTSwarm
Builds on: CodeAct BrowserGym SWE-Agent GPTSwarm
Closest papers by meaning
- OpenSage: Self-programming Agent Generation Engine (ICML 2026 · similarity 0.87)
- Agent S: An Open Agentic Framework that Uses Computers Like a Human (ICLR 2025 · similarity 0.86)
- OpenAgentSafety: A Comprehensive Framework For Evaluating Real-World AI Agent Safety (ICLR 2026 · similarity 0.86)
- OpenCUA: Open Foundations for Computer-Use Agents (NeurIPS 2025 · similarity 0.86)
- Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Use (ICML 2026 · similarity 0.86)
- ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.