Research map / Large language models / Language, safety and interpretability
Correlated Errors in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measures error correlation across 350+ LLMs on two MMLU leaderboards and a resume-screening task, finds errors are substantially correlated (higher for more accurate and same-provider/architecture models), and shows effects on LLM-as-judge evaluation and simulated hiring markets.
Problem
Whether different LLMs make meaningfully different errors, given assumptions that diversity in data, architecture and providers mitigates homogeneity; relevant for LLM-as-judge and high-stakes multi-agent use like hiring.
Approach
Measure pairwise error correlation as the agreement rate on wrong answers (conditional on both models being wrong) across hundreds of leaderboard models, regress it on provider, architecture, size and accuracy, and propagate the correlations into LLM-as-judge and labor-market simulations.
Key results
- On HELM, model pairs agree on average about 60% of the time when both are wrong, versus 1/3 under random choice; on HuggingFace the mean is 0.423 vs 0.127 random.
- Regressions show same company, same architecture, similar size, and higher individual accuracy are associated with higher error agreement; features explain 34% to 62% of variance across datasets.
- With 20 firms each using a random LLM, around 20% of applicants are systemically excluded, versus near 0 under uniform random preferences.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use models whose wrong answers are not independent | LLM pairs on HuggingFace · HuggingFace | agreement rate when both models are wrong: 0.127 vs. 0.423 | Helped |
| Use models whose wrong answers are not independent | LLM pairs on Helm · Helm | agreement rate when both models are wrong: \frac13 vs. 0.6 | Helped |
| Use models from the same company | LLM pairs · HuggingFace | agreement rate when both models are wrong: +0.066 | Helped |
| Use models with the same base architecture | LLM pairs · HuggingFace | agreement rate when both models are wrong: +0.076 | Helped |
| Increase model 1 accuracy | LLM pairs · HuggingFace | agreement rate when both models are wrong: +0.014 | Helped |
| Increase model 2 accuracy | LLM pairs · HuggingFace | agreement rate when both models are wrong: +0.013 | Helped |
| Use models from the same company | LLM pairs · Helm | agreement rate when both models are wrong: +0.022 | Helped |
| Increase individual model accuracy | LLM pairs · Helm | agreement rate when both models are wrong: +0.055 for Acc. 1; +0.054 for Acc. 2 | Helped |
| Increase model 1 correlation with human scores | LLM pairs rating resumes · Resumes | correlation in residuals: +0.015 | Helped |
| Increase model 2 correlation with human scores | LLM pairs rating resumes · Resumes | correlation in residuals: +0.028 | Helped |
| Use the latest LLMs rather than a single LLM | Hiring markets · Resume-job matching | match probability for highest-rated applicants | Helped |
| Use the same LLM across firms rather than uniformly random firm preferences | Hiring markets · Resume-job matching | average applicant rank of match | Helped |
Surprising findings
- Some unrelated models (e.g., google/text-unicorn@001 and writer/palmyra-x-v3) agree on nearly all jointly wrong answers despite no known relationship.
- More accurate models have more correlated errors, even after conditioning on provider and architecture.
Limitations
- Metrics treat all incorrect answers identically and ignore answer closeness and question difficulty.
- Evaluation limited to multiple-choice leaderboards and one resume-screening task, with open-ended generation not measured.
- Resume ratings are subjective and compared against only 15-30 hand-labeled jobs/resumes.
Experimental setup
Datasets: MMLU HuggingFace Open LLM Leaderboard 2 responses HELM leaderboard responses Upwork job postings LiveCareer resumes public resume repository
Benchmarks: MMLU (HuggingFace Open LLM Leaderboard 2 subset) MMLU (HELM subset) Resumes (job-resume fit, 30 jobs x 60 resumes)
Metrics: agreement rate when both wrong agreement rate when either wrong overall agreement residual correlation with human labels accuracy inflation systemic exclusion rate match probability average applicant rank
Baselines: uniformly random agreement baseline same-LLM monoculture random LLM selection uniformly random firm preferences
Builds on: MMLU Holistic Evaluation of Language Models (HELM) Open LLM Leaderboard 2
Closest papers by meaning
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.91)
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.90)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.90)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.89)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.89)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.