Research map / Large language models / Language, safety and interpretability

WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 25 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces WildGuard, an open multi-task moderation model (fine-tuned Mistral-7B-v0.3) that classifies prompt harmfulness, response harmfulness, and refusal, trained on the 87K-item WildGuardTrain and evaluated on the human-annotated WildGuardTest.

Problem

Open safety moderation tools perform poorly on adversarial jailbreak prompts and on detecting refusals in model responses, lagging far behind GPT-4 and leaving no open multi-task moderator covering prompt harm, response harm, and refusal.

Approach

A single instruction-tuned open model trained on a balanced multi-task mix of synthetic vanilla and adversarial (WildTeaming-transformed) prompts, in-the-wild prompts, annotator-written data, and matched refusal/compliance responses, jointly predicting all three moderation labels.

Key results

• WildGuard reaches 88.9% total F1 on WildGuardTest prompt harmfulness and outperforms GPT-4 by 3.9% on adversarial prompts; its average on public prompt-harm benchmarks is 86.1 vs. GPT-4's 84.6.
• On refusal detection, WildGuard outperforms the best open baseline LibrAI-LongFormer-ref by 26.4% (total F1 88.6 vs. 70.1 for the ref model's total column, per the paper's claim) and sits within 4.1% of GPT-4 (92.4).
• In a chat-moderation jailbreak test with Tulu-2-dpo-7B, WildGuard cuts attack success rate from 79.8% to 2.4% with RTA rising only from 0.0% to 0.4%.

Measured effects reported in the paper

Surprising findings

• Existing open moderators such as Llama-Guard2 perform poorly on adversarial jailbreak prompts, and the OpenAI Moderation API scores very low on several response-harm benchmarks.
• Removing synthetic vanilla data does not hurt, and in some cases helps, adversarial prompt scores, while single-task models slightly beat multi-task on refusal detection with XSTest-Resp.

Limitations

• Much of the training data is synthetic and may not match real human inputs; in-the-wild coverage is limited in size.
• Harm taxonomy and refusal definitions are authors' choices and may not match other developers' goals; finer-grained harm category classification is not supported.
• The model can make incorrect judgments that let unsafe content pass when used in automated moderation.

Experimental setup

Datasets: WildGuardTrain WildJailbreak LMSYS-Chat-1M WildChat HH-RLHF Anthropic Red-Teaming AegisSafetyTrain Safety-Tuned Llamas XSTest

Benchmarks: WildGuardTest XSTest-Resp ToxicChat OpenAI Moderation AegisSafetyTest SimpleSafetyTests HarmBenchPrompt HarmBenchResponse SafeRLHF BeaverTails WildJailbreak eval set

Metrics: F1 ASR RTA

Baselines: Llama-Guard Llama-Guard2 Aegis-Guard-Defensive Aegis-Guard-Permissive MD-Judge BeaverDam HarmBench classifiers LibrAI-LongFormer-ref OpenAI Moderation API GPT-4

Builds on: WildTeaming WildJailbreak Llama-Guard2 XSTest

Closest papers by meaning

• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal (ICLR 2025 · similarity 0.91)
• Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions (ICML 2025 · similarity 0.90)
• Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs (NeurIPS 2026 · similarity 0.90)
• WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models (NeurIPS 2024 · similarity 0.90)
• On Calibration of LLM-based Guard Models for Reliable Content Moderation (ICLR 2025 · similarity 0.89)
• MoGU: A Framework for Enhancing Safety of LLMs While Preserving Their Usability (NeurIPS 2024 · similarity 0.89)