Research map / Multi-agent systems and games / LLM agents
Agent-as-a-Judge: Evaluate Agents with Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Agent-as-a-Judge, where agentic systems evaluate agentic systems step by step, and releases DevAI, a 55-task AI development benchmark with hierarchical requirements, showing closer alignment with human consensus than LLM-as-a-Judge.
Problem
Existing agent evaluation relies on final outcomes or costly manual human judgment, giving little feedback on intermediate development steps; code-generation benchmarks like HumanEval, MBPP and SWE-Bench don't cover full AI development workflows.
Approach
Use an agentic judge with modular tools (graph, locate, read, retrieve, ask) to inspect a developer agent's workspace and trajectory and decide each requirement, rather than a single LLM judging text; evaluate on DevAI tasks organized as requirement DAGs with preferences.
Key results
- Agent-as-a-Judge reaches 90.44% alignment rate with human consensus on OpenHands in black-box setting, versus 60.38% for LLM-as-a-Judge.
- Agent-as-a-Judge costs $30.58 and 118.43 minutes, about 2.29% of the cost and 2.36% of the time of Human-as-a-Judge (~$1297.50).
- Human evaluation found GPT-Pilot and OpenHands each satisfied about 29% of requirements (dependency-aware) and only one task was fully solved.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Agent-as-a-Judge instead of LLM-as-a-Judge | judging MetaGPT · DevAI, black-box | alignment rate: 84.15% -> 88.52% | Helped |
| Use Agent-as-a-Judge instead of LLM-as-a-Judge | judging GPT-Pilot · DevAI, black-box | alignment rate: 65.30% -> 83.88% | Helped |
| Use Agent-as-a-Judge instead of LLM-as-a-Judge | judging OpenHands · DevAI, black-box | alignment rate: 60.38% -> 90.44% | Helped |
| Use Agent-as-a-Judge instead of LLM-as-a-Judge | judging MetaGPT · DevAI, gray-box | alignment rate: 68.86% -> 92.07% | Helped |
| Use Agent-as-a-Judge instead of LLM-as-a-Judge | judging GPT-Pilot · DevAI, gray-box | alignment rate: 71.85% -> 86.61% | Helped |
| Use Agent-as-a-Judge instead of LLM-as-a-Judge | judging OpenHands · DevAI, gray-box | alignment rate: 70.76% -> 90.16% | Helped |
| Add graph to ask (ablation) | Agent-as-a-Judge · OpenHands judging, DevAI | alignment rate: 65.03% -> 75.95% | Helped |
| Add read to ask + graph (ablation) | Agent-as-a-Judge · OpenHands judging, DevAI | alignment rate: 75.95% -> 82.24% | Helped |
| Add locate to ask + graph + read (ablation) | Agent-as-a-Judge · OpenHands judging, DevAI | alignment rate: 82.24% -> 90.44% | Helped |
| Add retrieve to ask + graph + read + locate (ablation) | Agent-as-a-Judge · OpenHands judging, DevAI | alignment rate: 90.44% -> 90.16% | No clear effect |
Surprising findings
- Adding the retrieve module did not improve alignment for OpenHands, and memory and planning modules were detrimental and dropped from the final design.
- Majority vote of three human evaluators aligned more closely with consensus than any single evaluator, and Agent-as-a-Judge outperformed some individual human evaluators on OpenHands.
Limitations
- Small benchmark of 55 tasks and only three developer frameworks evaluated.
- Human consensus is treated as approximate ground truth; evaluators were authors.
- Proof-of-concept judge is not optimized; memory and planning modules hurt performance and error propagation is a concern.
Experimental setup
Datasets: DevAI
Benchmarks: DevAI
Metrics: alignment rate judge shift requirements met (independent) requirements met (dependency-aware) task solve rate precision-recall curves cost time
Baselines: LLM-as-a-Judge Human-as-a-Judge MetaGPT GPT-Pilot OpenHands
Builds on: LLM-as-a-Judge SWE-Bench MLE-Bench
Closest papers by meaning
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation (ICLR 2026 · similarity 0.90)
- Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis (ICLR 2026 · similarity 0.89)
- Agentick: A Unified Benchmark for General Sequential Decision-Making Agents (NeurIPS 2026 · similarity 0.89)
- Automated Design of Agentic Systems (ICLR 2025 · similarity 0.89)
- Harbor Adapters and Harbor-Mix: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation (NeurIPS 2026 · similarity 0.88)
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.