Research map / Large language models / Language, safety and interpretability

SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SORRY-Bench, a 44-category class-balanced benchmark (440 base + 8.8K linguistically mutated unsafe instructions) with a 7K-annotation human judge dataset, showing fine-tuned 7B judges match GPT-4-scale evaluators, and benchmarks 50+ LLMs on refusal.

Problem

Existing LLM safety refusal benchmarks use coarse and imbalanced safety taxonomies, overlook linguistic and formatting variation in prompts, and rely on expensive large LLM judges.

Approach

A fine-grained 44-class taxonomy with class-balanced unsafe instructions, 20 linguistic mutations of prompts, and a systematic meta-evaluation showing that fine-tuning small LLM judges on human annotations yields accurate and cheap refusal evaluation.

Key results

• Fine-tuned Mistral-7b-instruct-v0.2 judge reaches 81.0% agreement with humans, comparable to GPT-4o at 78.9% (few-shot 79.6%), at about 11s per evaluation pass versus about 260s for GPT-4o.
• Claude-2.1 has about 5% fulfillment rate on SORRY-Bench, while Mistral-7b-instruct-v0.1 reaches about 90%.
• Persuasion techniques raise fulfillment rates across all tested models (by 5-66%), while encoding and encryption mutations mostly reduce fulfillment because models output nonsense.

Measured effects reported in the paper

Surprising findings

• Encoding and encryption mutations usually lower measured fulfillment because models produce undecipherable output that counts as refusal, not because they are safer.
• Few-shot prompting or chain-of-thought sometimes hurts judge agreement, e.g., Llama-3-70b with CoT drops to 32.1% agreement.

Limitations

• Binary refusal/fulfillment scale only; does not measure degree of harmfulness.
• Does not cover jailbreak attacks, multi-category unsafe prompts, or over-refusal/helpfulness.
• Possible data contamination since the public dataset may enter training corpora; the human judge dataset has restricted access.

Experimental setup

Datasets: AdvBench HarmBench Do-Not-Answer SALAD-Bench ALERT StrongREJECT XSTest HEx-PHI JBB-Behaviors SimpleSafetyTests

Benchmarks: SORRY-Bench

Metrics: fulfillment rate Cohen's kappa agreement time cost per evaluation pass

Baselines: GPT-4o GPT-3.5-turbo Llama-3-70b-instruct Llama-Guard-2-8B MD-Judge WildGuard HarmBench Classifier Perspective API Keyword Match Bert-Base-Cased

Builds on: AdvBench HarmBench Do-Not-Answer SALAD-Bench WildGuard

Closest papers by meaning

• OR-Bench: An Over-Refusal Benchmark for Large Language Models (ICML 2025 · similarity 0.94)
• WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (NeurIPS 2024 · similarity 0.91)
• Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation (ICLR 2025 · similarity 0.90)
• Refusal Direction is Universal Across Safety-Aligned Languages (NeurIPS 2025 · similarity 0.90)
• SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.90)
• EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions (NeurIPS 2025 · similarity 0.90)