Research map / Large language models / Instruction and fine-tuning
Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper shows that open instruction-tuned LLaMA/Falcon models are unsafe, and that adding a few hundred safety-refusal instruction examples to Alpaca-style LoRA tuning reduces harmfulness without hurting benchmarks, while too many safety examples cause exaggerated refusals.
Problem
Open-source instruction-tuned LLMs trained only for helpfulness readily follow malicious or harmful instructions, and there is no open safety instruction-tuning dataset to address this.
Approach
Convert red-teaming questions from the Anthropic Red Team dataset into imperative instructions with GPT-3.5-turbo refusal responses, mix 100 to 2,000 such safety examples into 20k Alpaca instructions, and LoRA-tune LLaMA and Falcon to measure the safety-helpfulness trade-off, including exaggerated safety.
Key results
- Adding 500 to 1,000 safety instructions to 20k Alpaca data substantially reduces harmfulness scores, with the reward model score for the unsafe model at 1.22 for 500 safety examples versus higher for the no-safety baseline.
- Language benchmarks (BoolQ, OpenBookQA, PIQA) stay roughly flat across safety-tuned LLaMA 7B variants, e.g. BoolQ 77.16 for the base vs 77.09 at 2,000 added safety examples.
- The model with 2,000 safety examples responds to more than 50% of XSTest safe prompts with exaggerated safety refusals.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a small set of safety examples during instruction tuning (ablation) | LLaMA 7B fine-tuned on Alpaca · I-MaliciousInstructions, Q-Harm, I-CoNa, I-Controversial, and I-PhysicalSafetyUnsafe | harmfulness | Helped |
| Increase safety examples from the baseline to 500–1,000 examples (ablation) | LLaMA 7B fine-tuned on Alpaca · Safety evaluation datasets | harmfulness | Helped |
| Add safety examples during instruction tuning (ablation) | LLaMA 7B fine-tuned on Alpaca · BoolQ, OpenBookQA, PIQA, and AlpacaEval | benchmark performance | No clear effect |
| Add 2,000 safety instructions, increasing the safety-data amount (ablation) | LLaMA 7B fine-tuned on Alpaca · XSTest | exaggerated-safety response rate: >50% | Hurt |
| Train with safety questions rather than safety instructions (ablation) | LLaMA 7B fine-tuned on Alpaca · Malicious instructions | harmfulness | Hurt |
| Use mixed safety questions and instructions instead of safety questions alone (ablation) | LLaMA 7B fine-tuned on Alpaca · Safety evaluation datasets | harmfulness | Helped |
| Use 500 curated safety examples rather than HH-RLHF chat training data | LLaMA 7B instruction-tuned model · I-MaliciousInstructions | harmfulness reward score: 1.65 -> 1.22 | Helped |
Surprising findings
- Training on safety questions instead of instructions generalizes poorly, with models trained on questions still complying with unsafe instructions more often.
- Guanaco, trained on Open Assistant, is much safer than LLaMA (Alpaca) and refuses many instructions, though it still gives some unsafe answers.
Limitations
- Only up to 2,000 safety examples were tested, so the point at which safety becomes overwhelming is unknown.
- Evaluation datasets are small (40 to 178 instructions) and use limited phrasing variation.
- Uses the OpenAI content moderation API to judge outputs from a model trained on OpenAI completions, introducing bias.
Experimental setup
Datasets: Alpaca-cleaned Anthropic Red Team dataset HH-RLHF OpenAssistant
Benchmarks: I-MaliciousInstructions I-CoNa I-Controversial I-PhysicalSafety Q-Harm XSTest I-Alpaca AlpacaEval PIQA BoolQ OpenBookQA
Metrics: harmfulness reward model score OpenAI content moderation max score refusal rate win rate manual preference accuracy
Baselines: LLaMA (Alpaca) Falcon (Alpaca) Guanaco MPT-7b-chat
Builds on: Alpaca LoRA Anthropic Red Team dataset Alpaca-LoRA
Closest papers by meaning
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.91)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.89)
- Safety Pretraining: Toward the Next Generation of Safe AI (NeurIPS 2025 · similarity 0.89)
- Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models (ICLR 2025 · similarity 0.89)
- On Prompt-Driven Safeguarding for Large Language Models (ICML 2024 · similarity 0.88)
- MoGU: A Framework for Enhancing Safety of LLMs While Preserving Their Usability (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.