AI Paper Atlas

Research map / Label-efficient and robust learning / Recognition and long-tail

STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems

Akash Bonagiri, Gerard Janno Anderias, Saee Patil, Angelina Lai, Devang Borkar, Gezheng Kang, Ishant Gandhi, Setareh Rafatirad, Houman Homayoun

ICML 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Group-robust classification · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace majority vote with posterior expected credit (PEC) aggregationagent ranking on MT-Bench · MT-Bench human judgmentsmean rank std under annotator subsampling: 0.259 -> 0.197Improved
replace majority vote with PECaggregation under adversarial annotators · synthetic, 40% adversarial annotatorsMSE (x10^-3): 8.42 -> 5.54Improved
replace majority vote with PECaggregation under adversarial annotators · synthetic, 40% adversarial annotatorsKendall's tau: 0.977 -> 0.996Improved
use PEC instead of Dawid-Skene for label recoveryaggregation · synthetic stress testsMSE vs latent ground truth: 3.47 -> 5.54 at 40% adversarialHurt
use PEC instead of majority vote with only 3 labels per itemaggregation · synthetic, 3 labels per itemMSE (x10^-3): 4.78 -> 5.71Hurt
use PEC instead of majority vote when agent quality gaps are tightaggregation · synthetic, tight agent gapKendall's tau: 0.939 -> 0.953Improved
use PEC instead of majority vote in high-consensus datasetsaggregation · ConvAbuse and QAGSranking stability: 0.000 -> 0.000 mean rank stdNo clear effect
use PEC instead of majority vote with only two annotatorsaggregation · MSLRmean rank std: 0.503 -> 0.587Hurt
increase labels per item from 3 to 9 (ablation)all aggregation methods · synthetic annotation densityMSE (x10^-3): 4.78 -> 1.09 (MV); 5.71 -> 0.96 (PEC)Improved
vary MT-Bench pairwise-to-correctness mapping (tie as win, loss, or partial values) (ablation)PEC agent ranking · MT-Benchagent rankNo clear effect
fraction of strict or lenient annotators raised to 40% (ablation)majority vote · syntheticMSE (x10^-3): 1.93 -> 3.65 (strict); 1.77 -> 4.44 (lenient)Hurt
raise hard item rate from 0% to 20% (ablation)majority vote · syntheticMSE (x10^-3): 1.10 -> 2.62Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy