Research map / Large language models / AI and society
Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measures how strongly ~50 AI safety benchmarks correlate with a PCA-derived general capabilities score and training compute across dozens of chat and base models, finding many safety benchmarks are largely capability proxies (safetywashing risk).
Problem
AI safety benchmarks may mainly track general capabilities and training compute rather than distinct safety properties, enabling "safetywashing" where capability gains are presented as safety progress.
Approach
Extract the first principal component of 12 capabilities benchmarks as a capabilities score, then compute Spearman correlations between each safety benchmark's scores and this score (and training FLOPs) across many models; high correlation flags a benchmark as a capability proxy, low correlation as a potentially distinct safety measure.
Key results
- Alignment (MT-Bench) has a capabilities correlation of 78.7% and compute correlation of 79.8% for chat models, and TruthfulQA MC1 has 81.2% capabilities correlation.
- WMDP biosecurity, chemical, and cybersecurity scores are strongly anticorrelated with capabilities (about -81% to -87%), and jailbreak benchmarks such as HarmBench TAP show low correlation (-42.8%).
- RMS calibration error has only 20.1% accuracy correlation for chat models, while Brier score has 95.5%, suggesting Brier score mostly proxies accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase upstream model capabilities | Chat models evaluated on alignment benchmarks · MT-Bench and LMSYS Chatbot Arena | benchmark score correlation: MT-Bench: 78.7%; LMSYS Chatbot Arena: 62.1% | Helped |
| Increase upstream model capabilities | Chat models evaluated on ETHICS · ETHICS | benchmark score correlation: 82.2% | Helped |
| Increase upstream model capabilities | Chat models evaluated on MACHIAVELLI Utility · MACHIAVELLI Utility | safety-oriented utility score correlation: -49.9% | Hurt |
| Increase upstream model capabilities | Chat models evaluated on Sycophancy · Sycophancy | non-sycophantic score correlation: -66.8% | Hurt |
| Increase upstream model capabilities | Chat models evaluated on BBQ Ambiguous, CrowS-Pairs English, and Discrim-Eval · Bias benchmarks | benchmark score correlation: BBQ Ambiguous: -37.3%; CrowS-Pairs English: 28.5%; Discrim-Eval: 33.2% | No clear effect |
| Increase upstream model capabilities | Chat models evaluated on TruthfulQA MC1 · TruthfulQA MC1 | benchmark score correlation: 81.2% | Helped |
| Increase upstream model capabilities | Chat models evaluated on GPQA and QuALITY · GPQA and QuALITY | benchmark score correlation: GPQA: 77.7%; QuALITY: 88.8% | Helped |
| Use RMS Calibration Error instead of Brier Score | Calibration evaluations across language and vision models · MMLU and ImageNet | accuracy correlation: RMS calibration error: 20.1%, 15.2%; Brier Score: 95.5%, 98.5% | No clear effect |
| Evaluate adversarial robustness using traditional perturbation benchmarks | Language models · ANLI, AdvGLUE, and AdvGLUE++ | capabilities correlation: ANLI: 81.5%; AdvGLUE: 65.5%; AdvGLUE++: 45.8% | Helped |
| Evaluate robustness with jailbreak attacks | Language models · Human Jailbreaks, TAP, and GCG | capabilities correlation: Human Jailbreaks: -31.4%; TAP: -42.8%; GCG: -28.4% | Hurt |
| Evaluate vision robustness with natural adversarial examples rather than gradient-based PGD | Vision models · ImageNet-A and PGD on ImageNet | capabilities correlation: ImageNet-A: 97.9%; PGD on ImageNet: -41.8% | Mixed |
| Increase upstream model capabilities | Models evaluated on WMDP · Biosecurity, Chemical Security, and Cybersecurity | WMDP score correlation: Biosecurity: -87.5%; Chemical Security: -81.1%; Cybersecurity: -86.0% | Hurt |
Surprising findings
- Brier score for calibration correlates with accuracy at about 95-98%, so it mostly proxies capabilities rather than calibration.
- Transfer jailbreak benchmarks (Human Jailbreaks, TAP, GCG) are not correlated with capabilities, unlike older static adversarial benchmarks such as ANLI and AdvGLUE.
Limitations
- Correlations are computed over a limited set of roughly 26-27 models per setting, and the capabilities score depends on the chosen benchmark set.
- Low correlation does not establish benchmark quality; the analysis measures entanglement with capabilities only.
- Some safety benchmark scores were recomputed or sign-flipped, and some models' FLOPs were estimated.
Experimental setup
Datasets: LogiQA PIQA Hellaswag Winogrande COPA MedQA ARC Challenge MMLU MATH LAMBADA GSM8K BBH
Benchmarks: MT-Bench LMSYS Chatbot Arena ETHICS MACHIAVELLI Sycophancy BBQ CrowS-Pairs Discrim-Eval TruthfulQA HaluEval GPQA QuALITY
Metrics: Spearman correlation with capabilities score Spearman correlation with training FLOPs RMS calibration error Brier score
Baselines: MT-Bench LMSYS Chatbot Arena TruthfulQA BBQ CrowS-Pairs Discrim-Eval GPQA QuALITY ANLI AdvGLUE AdvGLUE++ HarmBench
Builds on: MT-Bench HarmBench WMDP TruthfulQA ETHICS
Closest papers by meaning
- Position: AI Safety Must Embrace an Antifragile Perspective (ICML 2025 · similarity 0.88)
- Position: Require Frontier AI Labs To Release Small "Analog" Models (NeurIPS 2025 · similarity 0.86)
- SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior (ICML 2025 · similarity 0.86)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.85)
- BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (NeurIPS 2024 · similarity 0.85)
- AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions (CVPR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.