Research map / Multi-agent systems and games / LLM agents
VeRO: A Harness for Agents to Optimize Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VeRO, an outer harness giving coding agents versioned snapshots, budget-gated evaluation, and structured traces to optimize target agent harnesses, plus the VeRO-Bench suite, and empirically compares optimizers across five tasks.
Problem
Coding agents acting as optimizers of other LLM agents' harnesses (code, prompts, tools, orchestration) lack a standardized, controlled benchmark and infrastructure; optimization without guardrails risks contamination and unbounded budget use.
Approach
Treat agent optimization as an open-ended coding task: an outer harness enforces git-based versioning, budget-gated evaluation, test-data isolation, and structured trace access so different coding-agent optimizers can be compared fairly on improving target agent harnesses.
Key results
- Best VeRO-Agent Default (Sonnet 4.5) reaches 0.61 average score vs 0.50 baseline, while GEPA reaches 0.54 average.
- Claude Code Pure averages 0.53 vs 0.55 for Claude Code with VeRO Tools, showing harness tools add gains.
- On TerminalBench-2, VeRO tools with budget 178 raise pass rate from 30.3% to 37.1%.
Surprising findings
- Reasoning-heavy tasks (GPQA, MATH) show little to no optimization gain across configurations, and budget ablation shows flat holdout performance there.
- Optimizers default to prompt modifications over 50% of the time, with entropy of change types dropping across phases.
Limitations
- Budget is counted in evaluation calls rather than tokens or API cost, introducing variance.
- Public API instability and possible reward hacking via leaked ground truth are not fully controlled.
- Small splits (e.g., MATH 59 train, GAIA 50 train) and N=3 iterations limit statistical power; GPQA best commits selected on train.
Experimental setup
Datasets: MATH TAU-Bench Retail GAIA SimpleQA GPQA Diamond TerminalBench-2 FACTS Search
Benchmarks: GAIA GPQA MATH TAU-Bench Retail SimpleQA FACTS Search TerminalBench-2
Metrics: accuracy lift over baseline variance across iterations runtime per sample pass rate
Baselines: Claude Code Pure GEPA VeRO-Agent Resources Only VeRO-Agent Orchestrator
Builds on: DSPy TextGrad GEPA ADAS AFlow
Closest papers by meaning
- EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.89)
- Towards Direct Evaluation of Harness Optimizers via Priority Ranking (NeurIPS 2026 · similarity 0.88)
- SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.86)
- When Parallelism Pays Off: Cohesion-Aware Task Partitioning for Multi-Agent Coding (NeurIPS 2026 · similarity 0.86)
- Credit-Budgeted ICPC-Style Coding: When LLM Agents Must Pay for Every Decision (ICLR 2026 · similarity 0.86)
- GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.