AI Paper Atlas

Research map / Large language models / Alignment and preferences

AutoMetrics: Approximate Human Judgments with Automatically Generated Evaluators

Michael J Ryan, Yanzhe Zhang, Amol Salunkhe, Yi Chu, Di Xu, Diyi Yang

ICLR 2026 · Large language models › Alignment and preferences · approach: LLM-as-judge and feedback data

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
generate and regress task-specific metrics (AutoMetrics) instead of a single LLM judgeGPT-4o-mini · EvalGenKendall tau: 0.161 -> 0.334Improved
AutoMetrics instead of LLM-JudgeQwen-3-32B · RealHumanEvalKendall tau: 0.025 -> 0.145Improved
AutoMetrics instead of LLM-JudgeGPT-4o-mini · CoGymKendall tau: 0.199 -> -0.034Hurt
combine generated metrics with existing MetricBank metrics (vs generated only) (ablation)AutoMetrics · CoGym and EvalGen (small out-of-distribution)Kendall tau: CoGym 0.433 -> 0.329; EvalGen 0.503 -> 0.474Hurt
add existing MetricBank metrics to generated metrics (ablation)AutoMetrics · HelpSteer2Kendall tau: 0.308 -> 0.387Improved
use metric cards instead of one-sentence descriptions in retrieval (ablation)AutoMetrics retrieval · EvalGenKendall tau: 0.427 -> 0.522 (k=20)Improved
retrieve more metrics (k=5 to k=20/30) (ablation)AutoMetrics retrieval · EvalGenKendall tau: 0.414 -> 0.522Improved
regress on n=5 metrics instead of single best (no regression) (ablation)AutoMetrics regression · SimpEvalKendall tau: 0.232 -> 0.275Improved
vary number of regressed metrics n (ablation)AutoMetrics regression · five tasksKendall tauMixed
increase training set size beyond about 80 samples (ablation)AutoMetrics · SimpEval, HelpSteer2, RealHumanEvalKendall tauNo clear effect
use AutoMetrics as GEPA optimization reward instead of verifiable rewardReAct agent on tau-airline · tau-bench airlineaccuracy: 0.680 -> 0.720No clear effect
GEPA optimization with AutoMetrics reward vs unoptimized agentReAct agent on tau-airline · tau-bench airlineaccuracy: 60% -> 0.720Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy