Research map / Large language models / Retrieval and knowledge
HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a Llama-Guard-3 teacher into a 435M DeBERTa-v3-large safety guard using HarmAug, a prefix-attack data augmentation that elicits harmful instructions from a jailbroken LLM.
Problem
Large safety guard models (7B+) are too costly in memory and latency to deploy alongside LLMs on mobile devices or for red-teaming, and naive distillation on limited harmful instruction data yields underperforming small students.
Approach
Prefill an affirmative prefix ("I have an idea for a prompt:") in the LLM's response after a few-shot harmful-instruction prompt so the safety-aligned LLM continues generating harmful instructions; generate harmful and refusal responses, label with the teacher, and distill into a small encoder.
Key results
- DeBERTa-v3-large (435M) + HarmAug reaches average F1 0.7357 and AUPRC 0.8362 across four benchmarks, the highest average AUPRC among evaluated models, with F1 comparable to 7-8B guards.
- On WildGuardMix the HarmAug student reaches F1 0.7576 vs Llama-Guard-3 0.6998, at about 0.6% FLOPs, 25% latency, 12% peak memory, and 26% monetary cost of Llama-Guard-3.
- In red-teaming, the HarmAug reward model reaches test reward 0.82 vs 0.99 for the Llama-Guard-3 oracle while halving GFlowNet runtime (9h 8m vs 17h 23m).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add HarmAug-generated data to DeBERTa training | DeBERTa-v3-large safety guard · Average across OAI, ToxicChat, HarmBench, and WildGuardMix | AUPRC: 0.7962 -> 0.8362 | Helped |
| Add HarmAug-generated data to DeBERTa training | DeBERTa-v3-large safety guard · Average across OAI, ToxicChat, HarmBench, and WildGuardMix | F1: 0.7274 -> 0.7357 | Helped |
| Use the affirmative prefix attack to generate harmful instructions (ablation) | HarmAug instruction generation · LLM harmful-instruction generation | instruction-generation success rate | Helped |
| Generate synthetic instructions with Gemma-1.1 Instruct instead of another tested LLM (ablation) | DeBERTa-v3-large with HarmAug · Average across four benchmarks | AUPRC: 0.8362 vs. 0.8194; 0.8122; 0.8223; 0.8207; 0.8213 | Mixed |
| Use DeBERTa-large instead of DeBERTa-xsmall as the student (ablation) | HarmAug safety guard · Average across four benchmarks | AUPRC: 0.7588 -> 0.8362 | Helped |
| Use DeBERTa-large instead of DeBERTa-small as the student (ablation) | HarmAug safety guard · Average across four benchmarks | AUPRC: 0.7950 -> 0.8362 | Helped |
| Use DeBERTa-large instead of DeBERTa-base as the student (ablation) | HarmAug safety guard · Average across four benchmarks | AUPRC: 0.8094 -> 0.8362 | Helped |
| Add empty responses to the synthetic dataset (ablation) | DeBERTa-v3-large trained with HarmAug · Average across four benchmarks | AUPRC: 0.7631 -> 0.8362 | Helped |
| Add empty responses to the synthetic dataset (ablation) | DeBERTa-v3-large trained with HarmAug · OAI | F1: 0.7629 -> 0.7236 | Hurt |
| Replace the Llama-Guard-3 reward model with DeBERTa + HarmAug | GFlowNet red-teaming prompt generator · Red-teaming prompt generation | Runtime: 17h 23m -> 9h 8m | Helped |
| Use DeBERTa + HarmAug instead of Llama-Guard-3 | Safety guard inference on WildGuardMix test split · WildGuardMix test split | Latency / token: 172.62 \si{\microsecond} -> 43.22 \si{\microsecond} | Helped |
Surprising findings
- Removing the affirmative prefix or using a prefix-injection prefix substantially lowers the rate at which the LLM produces harmful instructions, indicating the prefix attack is key to bypassing refusal.
- Smallest generator Gemma-1.1-2b-it gave the largest AUPRC gain, outperforming 7-8B generator backbones.
Limitations
- Performance gains diminish as synthetic dataset grows, attributed to independent sampling causing redundancy.
- FlashAttention cannot be applied to DeBERTa due to disentangled attention, limiting latency optimization.
- Teacher labels and generated data depend on Llama-Guard-3 and other LLMs; the prefix-attack vulnerability is a safety concern.
Experimental setup
Datasets: WildGuardMix OpenAI Moderation ToxicChat HarmBench CipherChat AdvBench
Benchmarks: OpenAI Moderation ToxicChat HarmBench WildGuardMix CipherChat jailbreak attacks (GCG, PAIR, AutoDAN, Adaptive)
Metrics: F1 AUPRC FLOPs per token latency per token peak memory monetary cost test reward diversity
Baselines: EDA GFN Llama-Guard-1 Llama-Guard-2 Llama-Guard-3 WildGuard Aegis-Guard RoBERTa-R4 HateBERT OpenAI Moderation
Builds on: DeBERTa-v3 Llama-Guard-3 GFlowNet red-teaming prefix injection WildGuard
Closest papers by meaning
- RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards (NeurIPS 2025 · similarity 0.90)
- Safety Pretraining: Toward the Next Generation of Safe AI (NeurIPS 2025 · similarity 0.90)
- On Prompt-Driven Safeguarding for Large Language Models (ICML 2024 · similarity 0.89)
- Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy (ICLR 2025 · similarity 0.89)
- Look Before You Leap: Enhance Attention and Vigilance Regarding Harmful Content with GuidelineLLM (AAAI 2025 · similarity 0.88)
- WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.