Research map / Multi-agent systems and games / Games and equilibria

Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation

NeurIPS 2026 · Multi-agent systems and games › Games and equilibria · approach: Learning with strategic agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a principal-agent model of benchmark aggregation whose optimal item weights combine welfare alignment, improvability, and noise, and audits OLMES items on these three axes using WORKBank, EvoLM, and PolyPythias.

Problem

Benchmarks are summarized by uniform averaging of item scores, ignoring that labs strategically respond to the benchmark; the item-level value of benchmark items conditional on strategic lab investment is unclear.

Approach

Models benchmark aggregation as a linear-quadratic-CARA principal-agent game and derives a closed-form optimal weight v* = (M + rΣ)^{-1} M w combining welfare alignment w, improvability M, and seed-level noise Σ; then audits OLMES items on each primitive.

Key results

• Only about 6% of automation loading mass falls in the green-light welfare quadrant; math benchmarks (MathQA, AGIEval-AQuA, MMLU-Math) are Pareto-dominated and absent from the welfare frontier.
• SFT is the cheapest improvement axis for 50.3% of items, RL for 35.4%, and pretraining for 14.3%.
• HellaSwag shows about twice the binomial noise floor (Z≈0.62) on PolyPythia-410M, while MMLU, CSQA and MathQA fall below it (Z≈0.20).

Surprising findings

• Math benchmarks are welfare-dominated under the pro-worker operationalization, with no math items on the welfare Pareto frontier.
• Adversarially constructed completion benchmarks (HellaSwag, PIQA, ARC-Challenge) show excess seed noise beyond the binomial floor.

Limitations

• LLM-as-judge mapping of items to O*NET GWAs inherits LLM rater pathologies.
• Welfare functional reflects only worker preferences, excluding other stakeholders.
• Cost and noise panels (EvoLM 4B, PolyPythia 410M) are far below frontier scale; dollar costs are imputed estimates.

Experimental setup

Datasets: OLMES WORKBank O*NET Generalized Work Activities EvoLM PolyPythias MetaMathQA OpenMathInstruct-2 NuminaMath

Benchmarks: OLMES MMLU ARC-Challenge HellaSwag PIQA MathQA AGIEval-AQuA CommonsenseQA

Metrics: Pareto dominance welfare score per-item OLS improvability cost across-seed variance Spearman rank correlation

Baselines: uniform averaging of item scores

Builds on: multitask principal-agent framework OLMES EvoLM PolyPythias WORKBank

Closest papers by meaning

• AI Evaluation Should Require Standardized Item-Level Data Releases (NeurIPS 2026 · similarity 0.87)
• Establishing Best Practices in Building Rigorous Agentic Benchmarks (NeurIPS 2025 · similarity 0.87)
• Can We Trust Item Response Theory for AI Evaluation? (NeurIPS 2026 · similarity 0.86)
• JobBench: Aligning Agent Work With Human Will (NeurIPS 2026 · similarity 0.85)
• Impacts of Aggregation on Model Diversity and Consumer Utility (NeurIPS 2026 · similarity 0.85)
• Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks (ICML 2024 · similarity 0.84)