Research map / Large language models / Language, safety and interpretability

OR-Bench: An Over-Refusal Benchmark for Large Language Models

ICML 2025 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 9 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an automated pipeline that generates over-refusal prompts from LLM-rewritten toxic seeds, filtered by an LLM ensemble moderator, and releases OR-Bench (80K over-refusal, 1K hard, 600 toxic prompts) to measure over-refusal in 32 LLMs.

Problem

Safety alignment often causes LLMs to over-refuse benign prompts, but there is no large-scale way to find or measure prompts that should be answered yet get refused.

Approach

Generate toxic seeds with an uncensored LLM (Mixtral 8x7B), rewrite them into borderline-benign prompts with few-shot examples, and filter with an ensemble of three LLM moderators (GPT-4-turbo, Llama-3-70b, Gemini-1.5-pro) by majority vote, yielding a scalable over-refusal benchmark.

Key results

• Across 32 models from 8 families, safety (toxic rejection) and over-refusal are strongly correlated (Spearman 0.89), with most models trading over-refusal for safety.
• The ensemble moderator achieves 93% accuracy vs 94% for human experts on a 100-prompt moderation check, and beats all three individual human workers.
• Claude-2.1 rejects 99.8% of OR-Bench-Hard-1K prompts while Llama-3.1-70B rejects 3.0%; GPT-3.5-turbo-0125 rejects 12.7% of hard prompts vs 57.4% for GPT-3.5-turbo-0301.

Measured effects reported in the paper

Surprising findings

• Claude-3-opus, while rejecting many safe prompts, accepts only 1.9% of toxic prompts, and Claude-3 over-refusal is higher than its official report suggests.
• Using a system prompt that asks for helpful and safe responses substantially increases over-refusal, e.g. around 55% more benign prompts rejected for GPT-3.5-turbo-0125.

Limitations

• Some toxic prompts may pass LLM moderators undetected, and the moderators may be biased in favor of their own model families.
• The generation approach is one method and not claimed to be optimal for measuring over-refusal.
• Refusal is treated as a binary, ignoring nuanced responses that neither refuse nor fully comply.

Experimental setup

Datasets: AdvBench XSTest

Benchmarks: OR-Bench-80K OR-Bench-Hard-1K OR-Bench-Toxic AdvBench

Metrics: rejection rate acceptance rate Spearman correlation BERTScore accuracy

Baselines: XSTest AdvBench PHTest OpenAI moderation API

Builds on: XSTest AdvBench chain-of-thought prompting

Closest papers by meaning

• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal (ICLR 2025 · similarity 0.94)
• Refusal Direction is Universal Across Safety-Aligned Languages (NeurIPS 2025 · similarity 0.91)
• Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-Tuning (ICML 2025 · similarity 0.90)
• EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions (NeurIPS 2025 · similarity 0.90)
• Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.90)
• Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation (ICLR 2025 · similarity 0.89)