Research map / Large language models / Instruction and fine-tuning
Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that task-specific fine-tuning datasets can be reformatted (AOA, AutoIF) to raise harmfulness while staying benign-looking, and proposes Paraphrase, which rewrites safety data to mimic the user's prompt format, to restore safety with few mixed examples.
Problem
Task-specific fine-tuning (datasets with ground-truth answers) may compromise LLM safety, either accidentally for benign users or deliberately via benign-looking prompt structuring by malicious users, and existing safety-data mixing defenses from instruction-following work may not transfer.
Approach
Malicious users restructure task-specific data with prompting strategies (AOA, AutoIF, AutoIF+AOA) to increase harmfulness while evading toxicity filters; the defense paraphrases safety examples with an LLM to match the user data's format and style before mixing them in, closing the distribution gap that otherwise lets the model forget safety.
Key results
- Paraphrase keeps HarmI ASR at 0.00% for all four prompting strategies on LLaMA-2 7B at 50% mixing, while Base reaches 23.85% (AOA) and 12.12% (AutoIF+AOA) on GPT-3.5.
- Adversarial strategies raise HarmI ASR on LLaMA-2 7B up to 84.42% (HellaSwag, AutoIF+AOA) while at most 0.61% of data is flagged by OpenAI moderation.
- On GPT-3.5 PIQA, Paraphrase at 10% mixing achieves 0.00% HarmI ASR across strategies with 80-89% accuracy close to w/o Mixing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on GSM8K with the standard benign prompt | LLaMA-2 7B · GSM8K | accuracy: 19.11% -> 29.95% | Helped |
| Use adversarial task prompts (AOA, AutoIF, or AutoIF + AOA) instead of the benign prompt | LLaMA-2 7B fine-tuned on PIQA · PIQA | HarmI ASR: 0.00% -> 8.27%; 64.42%; 79.81% | Hurt |
| Mix safety data paraphrased to match the user-data format instead of no mixing | LLaMA-2 7B fine-tuned on mixed-prompt PIQA · PIQA (mixed prompting strategies) | HarmI ASR: 7.50% -> 0.00% | Helped |
| Mix safety data paraphrased to match the user-data format instead of Base mixing | LLaMA-2 7B fine-tuned on mixed-prompt PIQA · PIQA (mixed prompting strategies) | HarmQ ASR: 54.00% -> 4.00% | Helped |
| Mix safety data paraphrased to match the user-data format instead of no mixing | LLaMA-2 7B fine-tuned on mixed-prompt PIQA · PIQA (mixed prompting strategies) | accuracy: 74.32% -> 75.96% | Mixed |
| Mix safety data paraphrased to match the user-data format instead of Base mixing | GPT-3.5 fine-tuned on PIQA · PIQA | HarmI ASR: 23.85% -> 0.00% | Helped |
| Mix safety data paraphrased to match the user-data format instead of no mixing | GPT-3.5 fine-tuned on PIQA with Benign prompts · PIQA | accuracy: 86.89% -> 89.07% | Helped |
| Add AOA prompting rather than use the standard benign prompt | LLaMA-2 7B fine-tuned on BoolQ · BoolQ (B) | accuracy: 32.91% -> 64.10% | Helped |
| Use AutoIF + AOA prompts rather than the benign prompt | LLaMA-2 7B fine-tuned on HellaSwag · HellaSwag | HarmI ASR: 0.00% -> 84.42% | Hurt |
| Use Anti-AOA system prompt instead of the regular prompt (ablation) | LLaMA-2 7B fine-tuned on PIQA without safety-data mixing · PIQA | HarmI ASR: 8.27% -> 7.73% | Helped |
| Increase the number of fine-tuning epochs (ablation) | LLaMA-2 7B fine-tuned on PIQA with Benign or AOA prompts · PIQA | accuracy | Helped |
| Increase the number of fine-tuning epochs (ablation) | LLaMA-2 7B fine-tuned on PIQA with AutoIF or AutoIF + AOA prompts · PIQA | accuracy | Hurt |
Surprising findings
- Benign task-specific fine-tuning does not increase harmfulness (0% HarmI ASR), unlike instruction-following fine-tuning.
- Base mixing actually increased HarmI ASR for the multi-strategy PIQA mixture (27.88%) relative to w/o Mixing (7.50%).
Limitations
- Adversarial prompting strategies are demonstrative and easily detectable via structural analysis.
- Paraphrase requires converting the safety dataset for every user fine-tuning request, which is resource-intensive.
- Paraphrase opens a new attack vector where fine-tuning examples target the paraphrasing process.
Experimental setup
Datasets: BoolQ GSM8K HellaSwag MMLU OpenBookQA PIQA WinoGrande Harmful Instructions Harmful Questions XSTest safety fine-tuning dataset from prior work Alpaca/Dolly (referenced)
Benchmarks: BoolQ GSM8K HellaSwag MMLU OpenBookQA PIQA WinoGrande Harmful Instructions Harmful Questions XSTest
Metrics: attack success rate accuracy refusal rate toxicity detection rate
Baselines: Base safety data mixing Longest safety data mixing w/o Mixing Anti-AOA prompt Benign prompting
Builds on: Qi et al. fine-tuning compromises safety Absolutely Obedient Agent attack Base safety data mixing
Closest papers by meaning
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.92)
- Learning and Forgetting Unsafe Examples in Large Language Models (ICML 2024 · similarity 0.91)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.91)
- Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation (NeurIPS 2025 · similarity 0.91)
- What Makes and Breaks Safety Fine-tuning? A Mechanistic Study (NeurIPS 2024 · similarity 0.90)
- Safe LoRA: The Silver Lining of Reducing Safety Risks when Finetuning Large Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.