Research map / Multi-agent systems and games / LLM agents

Harbor Adapters and Harbor-Mix: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Ports over 80 agentic benchmarks to a unified Harbor adapter framework, runs a large evaluation of 8 models under two harnesses across 54 benchmarks, and curates Harbor-Index, an 82-task difficult meta-dataset.

Problem

Agentic benchmarks are heterogeneous in environments, interaction protocols, and scoring, so evaluating m benchmarks with n agents requires O(mn) integrations, limiting reliable, comparable, and affordable agent evaluation.

Approach

Decouple agents from benchmarks via a shared task schema (instruction, environment, tests, solution) so each benchmark and each agent needs only one integration (O(m+n)), validated by parity experiments and multi-stage audits; then distill a compact, difficult, audited subset (Harbor-Index).

Key results

• GPT-5.5 with Codex reaches 28.0% pass rate on Harbor-Index 1.0, the highest; no model-harness configuration exceeds 30%.
• Model fixed-effect range (0.451) is about 5.2x the harness range (0.087) in a linear mixed model, so base model matters more than harness.
• Frontier models use 42% of weaker-model tokens on the easiest difficulty bucket, but higher per-token prices mean they still cost more per trial.

Surprising findings

• Across more than 30 benchmarks, roughly one third of the hardest candidate tasks were rejected as broken rather than genuinely difficult.
• Native harnesses cut the timeout rate from 42% to 26% on matched tasks versus Terminus-2.

Limitations

• Adapter availability limits the large-scale evaluation to 54 of the collected benchmarks and a focused set of models.
• Empirical task difficulty is a configuration-dependent proxy that may be distorted by verifier errors and setup.
• Stronger future agents may exploit benchmark loopholes, so Harbor-Index must be maintained as a live benchmark.

Experimental setup

Datasets: SWE-bench Terminal-Bench HLE GPQA-Diamond GAIA2 SWE-bench Pro CRustBench AlgoTune GSO SimpleQA KUMO

Benchmarks: Harbor-Index 1.0 HLE GAIA GPQA Diamond AlgoTune GSO SWE-bench Verified LiveCodeBench BigCodeBench FinanceAgent LAB-Bench

Metrics: pass rate cost per run token usage Spearman rank correlation ICC failure-mode prevalence

Baselines: Terminus-2 Codex Claude Code Gemini CLI

Builds on: Terminal-Bench Harbor BenchPress

Closest papers by meaning

• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (NeurIPS 2024 · similarity 0.91)
• PACE: A Proxy for Agentic Capability Evaluation (NeurIPS 2026 · similarity 0.90)
• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition (ICML 2026 · similarity 0.89)
• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.89)
• UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.89)
• MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools (AAAI 2026 · similarity 0.89)