Research map / Multi-agent systems and games / LLM agents
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TerminalWorld, a data engine that reverse-engineers executable, test-verified terminal tasks from 80,870 asciinema recordings and benchmarks frontier LLMs and terminal agents on a 200-task verified subset.
Problem
Existing terminal-agent benchmarks are manually curated, costly to scale, and skew toward adversarial puzzles rather than authentic developer workflows.
Approach
Automatically reverse-engineer in-the-wild asciinema terminal recordings into instructions, reference solutions, Docker environments, and execution-calibrated test suites (with AllPassing, Nop, and Partial trials), making tasks authentic by construction.
Key results
- Best model Claude Opus 4.7 reaches 62.5% pass rate on TerminalWorld-Verified; all models range 49.0%-62.5%.
- TerminalWorld-Verified correlates only weakly with Terminal-Bench 2.0 scores (Pearson r=0.20).
- Median command-set overlap between agent and human reference solutions is only 21.4%.
Surprising findings
- Resource consumption is negatively correlated with success (turns r=-0.49, tokens r=-0.62), an 'efficiency paradox'.
- Open-weight models such as Kimi K2.6 outperform GPT-5.5 and Gemini 3.1 Pro on TerminalWorld-Verified despite lower Terminal-Bench 2.0 scores.
Limitations
- Scope limited to pure CLI tasks; TUI-based interactions excluded.
- Roughly 10% of tasks use end-of-life base images that break third-party agent installation, inflating error rates.
- Verified subset of 200 tasks was manually reviewed by four authors; full set of 1,530 is not individually verified.
Experimental setup
Datasets: asciinema terminal recordings Terminal-Bench
Benchmarks: TerminalWorld-Verified Terminal-Bench 2.0
Metrics: pass rate resolved rate cost per pass Pearson correlation command-set Jaccard overlap
Baselines: Terminus-2 mini-SWE-agent OpenHands Claude Code Codex CLI Gemini CLI Terminal-Bench 2.0
Builds on: Terminal-Bench Harbor harness Terminus-2
Closest papers by meaning
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (ICLR 2026 · similarity 0.92)
- Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments (ICML 2026 · similarity 0.88)
- On Data Engineering for Scaling LLM Terminal Capabilities (NeurIPS 2026 · similarity 0.87)
- Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents (NeurIPS 2026 · similarity 0.86)
- SETA: Scaling Environments for Terminal Agents (NeurIPS 2026 · similarity 0.86)
- AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.