Research map / Large language models / Language, safety and interpretability
WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mines 105K in-the-wild jailbreak tactics from chatbot logs to automatically compose diverse adversarial attacks (WildTeaming) and uses them to build the 262K-example WildJailbreak safety training dataset, showing that balanced vanilla/adversarial and harmful/benign data improves safety without hurting general capabilities.
Problem
Existing red-teaming methods find few diverse successful jailbreaks and open safety training data lacks adversarial complexity and benign contrast examples, leading to weak robustness and exaggerated refusals.
Approach
Mine jailbreak tactics from real user-chatbot logs (LMSYS-1M, WildChat) with GPT-4 and compose random subsets of them onto vanilla harmful prompts using an off-the-shelf attacker LLM with off-topic and low-risk pruning, then use the resulting attacks, paired with benign contrast queries, as safety SFT data.
Key results
- WildTeaming improves over PAIR by 4.6-25.6 points in ASR x5_30 across models while using fewer queries (3.8-5.5 points lower Query x5_30).
- Tulu2Mix+WildJailbreak achieves an 85.1% relative improvement on HarmBench over the Tulu2Mix baseline (ASR 20.8 to 3.1) and improves DoAnythingNow defense by 71.9% (ASR 49.7 to 14.0).
- Scaling WildJailbreak safety data mixed with 150K Tulu2Mix examples is needed (up to ~60K) to reach over 95% satisfactory response rate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compose mined in-the-wild tactics into attacks rather than use PAIR | WildTeam · HarmBench test set; Vicuna 7B | ASR×5₃₀: 59.5 -> 88.1 | Helped |
| Add off-topic and low-risk pruners (ablation) | WildTeam · HarmBench test set | attack performance | Helped |
| Do not fix the “seed leading sentence” tactic (ablation) | WildTeam · HarmBench test set | ASR×5₃₀: 80.5 vs. 56.1 | Hurt |
| Add WildJailbreak training data | Llama2 7B fine-tuned on T2M-no-refusal · HarmBench | ASR: 20.8 -> 3.1 | Helped |
| Add WildJailbreak training data | Llama2 7B fine-tuned on T2M-no-refusal · DoAnythingNow | ASR: 49.7 -> 14.0 | Helped |
| Add WildJailbreak training data | Llama2 7B fine-tuned on T2M-no-refusal · MT-Bench | total score: 5.84 -> 6.29 | Helped |
| Train on harmful data only instead of all four WildJailbreak components (ablation) | Llama2 7B fine-tuned on T2M-no-refusal · WildJailbreak test set; benign | RTA: 1.6 -> 54.4 | Hurt |
| Add benign examples to harmful-only training (ablation) | Llama2 7B fine-tuned on T2M-no-refusal · WildJailbreak test set; benign | RTA: 54.4 -> 1.6 | Helped |
| Train on vanilla data only instead of all four WildJailbreak components (ablation) | Llama2 7B fine-tuned on T2M-no-refusal · WildJailbreak test set | accuracy: 98.4 -> 70.7 | Hurt |
| Add adversarial training data to vanilla-only training (ablation) | Llama2 7B fine-tuned on T2M-no-refusal · WildJailbreak test set | accuracy: 70.7 -> 98.4 | Helped |
| Train on adversarial data only instead of all four WildJailbreak components (ablation) | Llama2 7B fine-tuned on T2M-no-refusal · WildJailbreak test set | accuracy: 98.4 -> 97.4 | Hurt |
| Add vanilla training data to adversarial-only training (ablation) | Llama2 7B fine-tuned on T2M-no-refusal · vanilla and adversarial safety benchmarks | safety performance | Mixed |
Surprising findings
- Training only on public safety data (HH-RLHF, Safe-RLHF, DAN) gave mediocre results compared with WildJailbreak, which the authors attribute to pairwise preference responses being poor SFT targets.
- Training only on harmful data, without benign contrast examples, leads to exaggerated safety (over-refusal of benign queries).
Limitations
- Tactics are limited by source data (LMSYS-1M and WildChat) and may not capture the full range of user jailbreak strategies.
- Synthetic adversarial prompts generated by Mixtral, GPT-3.5 and GPT-4 may not resemble in-the-wild human-written attacks in style or content.
- The WildJailbreak release is gated behind a content warning and terms agreement.
Experimental setup
Datasets: LMSYS-Chat-1M WildChat Tulu2Mix HH-RLHF Safe-RLHF Safety-Tuned Llamas HarmBench
Benchmarks: HarmBench AlpacaEval MT-Bench ToxiGen XSTest JailbreakTrigger DoAnythingNow WildJailbreak test MMLU GSM BBH TydiQA
Metrics: ASR Query PPL ASR x5 30 Query x5 30 Sim #Tactic RTA F1 win rate
Baselines: PAIR AutoDAN GCG Tulu2Mix Tulu2Mix-public-safety
Builds on: HarmBench PAIR Tulu 2 Safety-Tuned Llamas XSTest
Closest papers by meaning
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.91)
- Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions (ICML 2025 · similarity 0.91)
- MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks (NeurIPS 2026 · similarity 0.90)
- WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (NeurIPS 2024 · similarity 0.90)
- Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models (ICLR 2025 · similarity 0.89)
- Metis: Learning to Jailbreak LLMs via Self-Evolving Metacognitive Policy Optimization (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.