Research map / Large language models / Language, safety and interpretability
Learning and Forgetting Unsafe Examples in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows aligned LLMs learn unsafe data during downstream finetuning and forget unsafe examples more than others during safety finetuning, then proposes ForgetFilter (FF), which filters unsafe examples by their forgetting rate.
Problem
Released LLMs finetuned by downstream users on noisy custom data can learn unsafe content (bias, toxicity, harmfulness), and safety finetuning afterwards causes catastrophic forgetting of useful downstream knowledge.
Approach
Unsafe examples are forgotten faster than other data during safety finetuning in larger LMs; ForgetFilter exploits this by training a model on noisy data, then finetuning on safe data, and discarding examples with high forgetting rate before retraining from the initial checkpoint.
Key results
- FF reduces bias score to 0.08 and toxicity to 0.11 versus 0.57 and 0.45 for downstream-only finetuning, while keeping SQuAD F1 at 83.1 (vs 82.4 without safety measures).
- Sequential safety finetuning (SafetyFT) lowers bias/toxicity to near zero but drops downstream F1 from 82.4 to 75.7 on bias.
- FF filtering F1 reaches 82.3-91.1 for bias, 81.2-86.3 for toxicity, and 68.7-73.4 for harmfulness across unsafe proportions of 25-75%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| filter unsafe examples using forgetting rate instead of applying moral self-correction alone | LLaMA-7B safe customized finetuning · Bias | bias score: 0.10 -> 0.08 | Helped |
| filter unsafe examples using forgetting rate instead of replay | LLaMA-7B safe customized finetuning · Toxicity | toxicity score: 0.43 -> 0.11 | Helped |
| apply forgetting-based filtering rather than sequential safety finetuning | LLaMA-7B downstream finetuning · Bias | F1: 75.7 -> 83.1 | Helped |
| combine forgetting-based filtering with moral self-correction | LLaMA-7B safe customized finetuning · Bias | bias score: 0.08 -> 0.07 | Helped |
| combine forgetting-based filtering with moral self-correction rather than filtering alone | LLaMA-7B safe customized finetuning · Toxicity | toxicity score: 0.11 -> 0.09 | Helped |
| use forgetting-based filtering instead of sequential safety finetuning | LLaMA-7B safe customized finetuning · Toxicity | F1: 68.1 -> 77.8 | Helped |
| reduce safe examples used during filtering from 7000 to 3000 (ablation) | FF · Bias, toxicity, and harmfulness filtering | F1: around 1.2% | Hurt |
| use safe examples from a different safety category than the unsafe examples (ablation) | sequential safety finetuning · Toxicity forgetting | forgetting rate: around 60% -> around 19% | Hurt |
| filter unsafe examples with FF across different unsafe-data proportions | FF · Bias filtering, 25% unsafe examples | F1: 82.3 | Helped |
| increase unsafe-data proportion from 25% to 75% (ablation) | FF · Bias filtering | F1: 82.3 -> 91.1 | Helped |
| increase unsafe-data proportion from 25% to 75% (ablation) | FF · Toxicity filtering | F1: 81.2 -> 86.3 | Helped |
| increase unsafe-data proportion from 25% to 75% (ablation) | FF · Harmfulness filtering | F1: 68.7 -> 73.4 | Helped |
Surprising findings
- Unsafe examples are forgotten more than safe and task data during safety finetuning, but only in models larger than GPT-2 M; the effect is absent in the smallest model.
- Recalled unsafe knowledge: after safety finetuning, models quickly regain bias in subsequent downstream finetuning sessions and become even more biased over time in interleaved training.
Limitations
- FF requires a set of safe examples whose distribution must cover the unsafe types; novel unsafe content may be missed.
- Threshold phi is set manually and automatic selection is left for future work.
- Harmfulness has no automated metric, so safety finetuning results omit it.
Experimental setup
Datasets: BBQ Pile toxicity subsample (Detoxify scores) HarmfulQA SQuAD Alpaca
Benchmarks: BBQ Pile toxicity SQuAD
Metrics: bias score Detoxify toxicity score F1 ROUGE-1 forgetting rate filtering F1
Baselines: SafetyFT (sequential safety finetuning) example replay moral self-correction
Builds on: LoRA chain-of-thought prompting zero-shot CoT forgetting-based noisy label filtering
Closest papers by meaning
- Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models (ICLR 2025 · similarity 0.91)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.90)
- How Much Can We Forget about Data Contamination? (ICML 2025 · similarity 0.90)
- Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack (ICML 2025 · similarity 0.90)
- Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy (NeurIPS 2025 · similarity 0.90)
- Safety Pretraining: Toward the Next Generation of Safe AI (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.