Research map / Large language models / AI and society

SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety

AAAI 2025 · Large language models › AI and society · approach: Critiques of LLM claims and judging · 15 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematic, community-driven review of 144 open LLM safety datasets (2018–2024) covering purposes, formats, creation methods, languages, licensing, and their use in model release reports and popular benchmarks.

Problem

LLM safety datasets are numerous, fragmented and fast-growing, making it hard for researchers to find relevant ones and to see coverage gaps and how they are used in practice.

Approach

A first systematic review of open LLM safety datasets, built via iterative community solicitation plus snowball search, with structured coding of 23 attributes per dataset, complemented by an analysis of safety datasets used in model release publications and popular benchmarks.

Key results

• Reviewed 144 open LLM safety datasets published June 2018 to December 2024; 59 published in 2024 and 43 in 2023.
• Of 29 model release publications examined, 16 report some safety evaluation and only 14 distinct open safety datasets are used in total; TruthfulQA is used in 8 of these 16.
• 113 of 144 datasets (78.5%) are exclusively English; 21 of 102 datasets released since 2023 are fully synthetic.

Surprising findings

• Of 29 model release publications, 11 of the 16 that report safety evaluations use undisclosed proprietary data, and Llama 3, Qwen 2 and Phi 3 report results only on proprietary safety datasets.
• TruthfulQA, the most used open safety dataset in model releases, is often framed by developers as a capability rather than a safety evaluation.

Limitations

• The review likely misses some relevant datasets despite community-driven search.
• Quality of datasets is not judged, since utility is context dependent.
• Model release analysis covers only 29 models from the top of the LMSYS leaderboard as of July 2024.

Experimental setup

Datasets: TruthfulQA BBQ AnthropicRedTeam ToxiGen RealToxicityPrompts WildChat XSTest ETHICS DoNotAnswer HarmBench DecodingTrust SimpleSafetyTests

Benchmarks: TrustLLM HELM Safety LLM Safety Leaderboard HELM Classic HELM Instruct Open LLM Leaderboard RewardBench Eleuther AI Evaluation Harness BIG-Bench

Metrics: dataset counts frequency of dataset use in model release publications frequency of dataset use in benchmarks

Closest papers by meaning

• RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios? (AAAI 2025 · similarity 0.92)
• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.91)
• CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models (ICML 2025 · similarity 0.91)
• MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.91)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.90)
• Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.90)