Research map / Multi-agent systems and games / LLM agents
Towards Direct Evaluation of Harness Optimizers via Priority Ranking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes priority ranking, a low-cost step-level evaluation where harness optimizers rank harness components by expected impact, and the SHOR dataset of 182 human-verified scenarios to enable it.
Problem
Harness optimizers (LLM agents that iteratively update a target agent's harness) are evaluated only by end-to-end improvement of the target agent, which hides erroneous intermediate update actions and is expensive to run.
Approach
Evaluate optimizers directly by asking them to rank harness components (prompt, memory, workflow, tool) by their potential to improve or hinder agent performance when updated; ground-truth rankings come from measured success-rate changes of component-targeted variant harnesses proposed by coding-agent annotators, with Kendall's W consistency filtering and human verification.
Key results
- Priority ranking Acc@1 correlates with optimizers' target-agent SR improvement in harness optimization (Pearson rho = 0.602 in-domain, p = 0.038; 0.632 out-of-domain per the paper's summary).
- Best average Acc@1 is 0.305 (OpenHands-CLI with DeepSeek-V4-Pro); best average NDCG is 0.793 (Claude Code with Sonnet 4.6), versus random 0.250 Acc@1.
- Priority ranking costs $63.7 and 63.2 min versus $524.3 and 1089.8 min for end-SR observation (about 8x cheaper, 17x faster).
Surprising findings
- Optimizers' predictions of whether their own harness update improves agent performance are near or below random (GPT-5.2 0.568, Qwen3.6-Plus 0.432, Sonnet 4.6 0.386).
- Good agent harnesses are not necessarily good optimizer harnesses: GPT-5.2 under OpenHands-CLI outperforms GPT-5.2 under Codex in 3 of 4 domains.
Limitations
- Only four harness component categories (prompt, tool, memory, workflow), which may underestimate sensitivity to finer changes.
- Single target agent evaluated; findings may not transfer to multi-agent systems.
- Labels bounded by updates annotator agents can produce; construction costs about $4,000 per domain and retains only about 30% of candidates.
Experimental setup
Datasets: Spider 2.0-lite tau^2-Bench GAIA SWE-bench Verified AppWorld GPQA
Benchmarks: SHOR Spider 2.0-lite tau^2-Bench GAIA SWE-bench Verified AppWorld GPQA
Metrics: Acc@1 NDCG Pearson correlation cost time resolved rate
Baselines: Random Human performance
Builds on: ReCreate Meta-Harness GEPA ADAS
Closest papers by meaning
- VeRO: A Harness for Agents to Optimize Agents (ICML 2026 · similarity 0.88)
- Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs (NeurIPS 2024 · similarity 0.84)
- DesignX: Human-Competitive Algorithm Designer for Black-Box Optimization (NeurIPS 2025 · similarity 0.83)
- Improving Parallel Program Performance with LLM Optimizers via Agent-System Interfaces (ICML 2025 · similarity 0.83)
- FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics (NeurIPS 2026 · similarity 0.83)
- EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.