Research map / Multi-agent systems and games / LLM agents

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Terminal-Bench 2.0, a 89-task (text says 74 in some places) hard benchmark of containerized terminal tasks, evaluates 16 frontier and open models across agent scaffolds, and analyzes failure modes.

Problem

Existing agent benchmarks either do not measure realistic tasks or are not difficult enough for frontier models; there is a need for hard, realistic, long-horizon terminal tasks to evaluate agents.

Approach

A crowd-sourced, expert-verified benchmark of containerized terminal tasks (instruction, Docker image, tests, oracle solution) run through a unified harness (Harbor) with a neutral scaffold (Terminus 2), plus a trajectory and command-level failure taxonomy and an audit pipeline including an adversarial exploit agent.

Key results

• Codex CLI with GPT-5.2 achieves the highest resolution rate of 62.9% on Terminal-Bench 2.0.
• Terminus 2 with Claude Opus 4.5 reaches 57.8% and with Gemini 3 Pro 56.9%; smaller models such as GPT-OSS-20B score about 3%.
• Command error rates range from 9.2% (Grok 4) to 26.7% (GPT-OSS-120B); missing executables are the most frequent command failure (24.1%).

Surprising findings

• Model selection matters more than agent scaffold for performance, per the authors' analysis.
• Higher token usage does not correlate with better performance, and turn count shows essentially no correlation with success.

Limitations

• No private held-out test set; tasks are publicly hosted, so contamination remains possible.
• Some non-determinism from hardware, external APIs, and infrastructure; no quantitative analysis of impact of benchmark flaws.
• Agents have internet access, so cheating by reading oracle solutions is theoretically possible.

Experimental setup

Datasets: Terminal-Bench 2.0 tasks (89 tasks, crowd-sourced from 93 contributors)

Benchmarks: Terminal-Bench 2.0

Metrics: resolution rate token usage command error rate failure mode distribution

Baselines: Claude Code Codex CLI Gemini CLI OpenHands Mini-SWE-Agent Terminus 2

Builds on: Harbor Terminus 2 MAST Docent Agentic Benchmark Checklist

Closest papers by meaning

• TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks (NeurIPS 2026 · similarity 0.92)
• SWE-Marathon: Can AI Agents Autonomously Complete Ultra-Long-Horizon Software Work? (NeurIPS 2026 · similarity 0.90)
• Towards a Science of AI Agent Reliability (ICML 2026 · similarity 0.89)
• Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies (NeurIPS 2026 · similarity 0.89)
• Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments (ICML 2026 · similarity 0.88)
• Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents (NeurIPS 2026 · similarity 0.88)