Research map / Large language models / AI and society
Great Models Think Alike and this Undermines AI Oversight
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAPA, a chance-adjusted probabilistic model-similarity metric based on overlap in model mistakes, and uses it to show LLM judges favor similar models, weak-to-strong gains depend on complementary knowledge, and frontier model errors are becoming more correlated.
Problem
AI oversight (LM-as-a-judge and training on LM annotations) may be undermined if models are similar, but there was no good functional similarity metric that adjusts for accuracy and uses probabilities, and no evidence on how similarity affects oversight.
Approach
CAPA extends error consistency (Cohen's kappa-style chance adjustment for accuracy) by using output probabilities for observed agreement and a chance term that uses each model's mean probability on the correct option with uniform spreading over incorrect options, so that MCQ options need not be permuted-class-aligned.
Key results
- LLM judges' scores correlate positively with CAPA similarity across all judges (average Pearson r=0.84, p<0.01), even after controlling for accuracy (partial r 0.35-0.65).
- Across 12 weak-strong model pairs, similarity between weak supervisor and strong student inversely correlates with weak-to-strong gain (r=-0.85; r=-0.35 after controlling for accuracy gap).
- Across 130 OpenLLM Leaderboard 2 models, CAPA similarity rises strongly with capability percentile, indicating increasingly correlated mistakes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase CAPA similarity to the judge | Models evaluated by LLM judges on MMLU-Pro · MMLU-Pro | judgment score: average Pearson r=$0.84$ | Helped |
| Increase CAPA similarity to the judge while controlling for evaluated-model accuracy | Models evaluated by LLM judges on MMLU-Pro · MMLU-Pro | judgment score association | Helped |
| Use a more similar model rather than the more capable Llama-3.1-70B-Instruct | Qwen2.5-72B-Instruct judge · MMLU-Pro open-style answers | judgment score: 71\% ... 67\% | Helped |
| Reduce similarity between the weak supervisor and strong student | Weak-to-strong training · 15 binary classification NLP tasks | accuracy gain from training: $r=-0.85$; $r=-0.35, p < 0.01$ | Helped |
| Add complementary knowledge from the weak supervisor to ground-truth elicitation | Strong student model · 15 datasets, 12 model pairs | accuracy gap: $+11.2$ -> $+14.1$ | Helped |
| Increase model capability | Models from different developers · MMLU-Pro and Big Bench Hard (BBH) | CAPA similarity | Helped |
| Use instruction-tuned models | Models compared for error similarity · MMLU-Pro and BBH | CAPA similarity trend with capability | Helped |
Surprising findings
- Model mistakes become more correlated as capabilities increase, which the authors frame as a risk for AI oversight.
- Ceiling for weak-to-strong training estimated as union of weak supervisor and strong elicited correct predictions (+14.1 vs +11.2 for prior elicitation ceiling).
Limitations
- Establishing causation: correlation between similarity and oversight outcomes, not shown to be causal.
- Restricted to multiple-choice and classification tasks; free-text similarity not addressed.
- Uniform prior assumption over incorrect options is a modeling assumption.
Experimental setup
Datasets: MMLU-Pro Big Bench Hard AlpacaEval 15 binary NLP classification tasks from weak-to-strong setup OpenLLM Leaderboard 2 sample-wise predictions
Benchmarks: MMLU-Pro Big Bench Hard AlpacaEval
Metrics: CAPA (kappa_p) Pearson correlation partial correlation multiple regression judgment score accuracy weak-to-strong accuracy gain
Baselines: error consistency %Agreement %Flips Cohen's kappa Scott's pi Jensen-Shannon distance
Builds on: error consistency Cohen's kappa weak-to-strong generalization MMLU-Pro
Closest papers by meaning
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.87)
- Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models (NeurIPS 2024 · similarity 0.87)
- Model Spec Midtraining: Improving How Alignment Training Generalizes (NeurIPS 2026 · similarity 0.86)
- Debate Helps Weak-to-Strong Generalization (AAAI 2025 · similarity 0.86)
- Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment (NeurIPS 2025 · similarity 0.86)
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.