Research map / Large language models / Agents, code and math

PACE: A Proxy for Agentic Capability Evaluation

NeurIPS 2026 · Large language models › Agents, code and math · approach: Agent instruction-following benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PACE, which selects a 100-instance proxy from non-agentic benchmarks and fits a bootstrap least-squares regression to predict agentic benchmark scores (GAIA, SWE-Bench Verified/Multimodal, SWT-Bench) across 14 models at about 1/100 the cost.

Problem

Full agentic benchmark evaluations (SWE-Bench, GAIA) are expensive, slow and infrastructure-heavy, while cheap non-agentic benchmarks are not known to predict agentic performance reliably.

Approach

Select a compact subset of non-agentic instances by combining a target-relevance local signal (Spearman correlation with target scores) and a global SVD-leverage signal, then fit a noise-aware (bootstrapped) least-squares regression to predict target agentic scores and pairwise preferences.

Key results

• PACE-Bench with 100 instances achieves average LOOCV MAE 3.80%, Spearman 0.81 and pairwise accuracy 84.37% across four agentic targets.
• PACE outperforms Lasso (5.69% MAE, 0.46 Spearman) and Ridge (5.16% MAE, 0.76 Spearman) baselines under LOOCV.
• At matched prediction quality, PACE needs roughly 100x less evaluation cost than random target-sampling of the agentic benchmark.

Surprising findings

• Lasso and Ridge fit near-perfectly in-sample (up to 0.02% MAE for Lasso on GAIA) but degrade sharply under LOOCV, indicating overfitting in the small-model regime.
• SWE-Bench Verified proxy is dominated by multimodal and retrieval instances rather than direct code-editing instances.

Limitations

• Calibration set of only 14 models is small relative to the 100-dimensional feature space, so regression weights are noisy.
• Proxy gaming risk: a fixed public proxy set can be optimized against without genuine agentic gains.
• Evaluated only on four agentic benchmarks sharing the OpenHands harness; generalization to browser or embodied agents untested.

Experimental setup

Datasets: GAIA SWE-Bench Verified SWE-Bench Multimodal SWT-Bench ACPBench AIME 2025 BEIR (NFCorpus) BFCL DebugBench GPQA HumanEval IFEval

Benchmarks: GAIA SWE-Bench Verified SWE-Bench Multimodal SWT-Bench

Metrics: MAE Spearman correlation Pearson correlation pairwise ranking accuracy

Baselines: Lasso Ridge random target-sampling

Builds on: tinyBenchmarks Bradley-Terry model lm-evaluation-harness OpenHands Index

Closest papers by meaning

• Harbor Adapters and Harbor-Mix: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation (NeurIPS 2026 · similarity 0.90)
• AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.90)
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (NeurIPS 2024 · similarity 0.90)
• \$OneMillion-Bench: How Far are Language Agents from Human Experts? (NeurIPS 2026 · similarity 0.90)
• LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth (ICML 2026 · similarity 0.89)
• UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.89)