Research map / Large language models / Language, safety and interpretability

Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Uncertainty quantification for LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Total Evaluation Error (TEE), a variance-decomposition and D-study framework for LLM measurement pipelines that yields corrected SEs/CIs and ranks design changes (judges, prompts, items, replicates), validated on AILuminate, MMLU, and Chatbot Arena.

Problem

Standard confidence intervals for LLM evaluations and annotation ignore variance from judge model choice, temperature, and prompt phrasing, so they under-cover (worsening with more items) and leave benchmarks exposed to gaming via pipeline noise.

Approach

A linear mixed-model decomposition of LLM pipeline variance (items, prompts, judges, temperature, interactions, replicates) with a generalizability-theory D-study that projects SE under alternative designs, so pilots can produce honest CIs and pick the design change that most reduces error.

Key results

• Naive 95% CI coverage on Chatbot Arena drops to 79% at n_m=2,000 while TEE-corrected coverage stays at or above 95%.
• On MMLU, TEE-guided allocation halves naive RMSE at equal API budget and keeps 95% CI coverage at or above 98%.
• TEE-LLM averaging reduces best-of-K=27 Chatbot Arena gaming surface from 56 to 32 Elo; TEE Likert raises AB-only human-vote agreement by 7.9 pp (74.9% vs 67.0%).

Surprising findings

• Judge model choice contributes 43.8% of variance on AILuminate, while prompt wording contributes less than 0.5%.
• Naive CI coverage worsens as sample size grows, because omitted judge and prompt variance does not shrink with items.

Limitations

• Assumes additive two-way-interaction linear mixed model and normal random effects; variance shares on probability scale (LPM) rather than logistic link.
• Cannot assess whether items span the target construct (coverage error); reasoning-trace variance is absorbed into residual.
• Experiments limited to English and frontier models; high-ambiguity tasks untested.

Experimental setup

Datasets: AILuminate v1.0 DEMO prompts MMLU Chatbot Arena human preference (lmarena-ai/arena-human-preference-100k)

Benchmarks: AILuminate v1.0 MMLU Chatbot Arena

Metrics: standard error 95% CI coverage RMSE Elo gaming surface per-match agreement with human votes Spearman correlation

Baselines: naive single-judge pipeline standard allocation (no replication) naive allocation (replication) Chatbot Arena human leaderboard

Builds on: generalizability theory total survey error decision study (D-study) REML

Closest papers by meaning

• LiveBench: A Challenging, Contamination-Limited LLM Benchmark (ICLR 2025 · similarity 0.87)
• AI Evaluation Should Require Standardized Item-Level Data Releases (NeurIPS 2026 · similarity 0.87)
• Rethinking LLM Evaluation: Can We Evaluate LLMs with 200× Less Data? (ICLR 2026 · similarity 0.86)
• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.86)
• Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints (ICML 2025 · similarity 0.86)
• The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.86)