Research map / Large language models / Instruction and fine-tuning
Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TA-SFT, which adds a semantic EMD-based penalty (with a novel lower bound) or an NLCL likelihood penalty during SFT on a small set of self-generated unsafe responses to toxic prompts, to make LLMs safe with about 0.5% harmful data.
Problem
LLMs produce unsafe responses to toxic prompts; existing safety-at-SFT methods need large amounts of human data or safe responses generated by another LLM.
Approach
Penalize the next-token distribution of the model toward the unsafe tokens seen in a small set of harmful responses, using a semantic cost (cosine distance between token embeddings) inside an Earth Mover Distance term, optimized through a proposed squared-norm lower bound on EMD.
Key results
- EMD loss yields 0 harmful responses on all four benchmarks for Llama 7b with 1,000 toxic prompts and still keeps 0 on most with 100 prompts, while NLCL yields up to 42 harmful responses on I-CoNa with 100 prompts.
- On AlpacaEval, EMD scores 57.37% for llama 7b versus 56.14% for SFT and 54.48% for NLCL, and 73.35% for llama3.1 8b versus 72.05% for SFT.
- Contrastive augmentation with LLM-generated seemingly toxic samples makes Llama 7b less safe and can induce non-English responses as the penalty weight increases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EMD semantic penalty instead of NLCL with 100 harmful examples | TA-SFT on Llama 7b · I-CoNa | harmful responses: 42 -> 5 | Helped |
| Use EMD semantic penalty instead of NLCL with 100 harmful examples | TA-SFT on Llama 7b · I-Malicious | harmful responses: 6 -> 0 | Helped |
| Reduce harmful training examples from 1000 to 100 (ablation) | EMD TA-SFT on Llama 7b · I-CoNa | harmful responses: 0 -> 5 | Hurt |
| Reduce harmful training examples from 1000 to 100 (ablation) | NLCL TA-SFT on Llama 7b · I-CoNa | harmful responses: 0 -> 42 | Hurt |
| Use EMD rather than STL safe-response training with 100 examples | Llama 7b safety tuning · I-CoNa | harmful responses: 70 -> 5 | Helped |
| Replace EMD with NLCL | TA-SFT on Llama 7b · AlpacaEval | preference rate (%): 57.37 -> 54.48 | Hurt |
| Use EMD instead of standard SFT | Llama 7b response quality · AlpacaEval | preference rate (%): 56.14 -> 57.37 | Helped |
| Add LLM-generated contrastive samples for seemingly toxic prompts (ablation) | EMD and NLCL TA-SFT on Llama 7b · four harmfulness benchmarks | harmfulness score | Hurt |
| Increase penalty weight when training with contrastive samples (ablation) | EMD and NLCL TA-SFT on Llama 7b · language performance | non-English answers | Hurt |
| Train further with EMD or NLCL (ablation) | TA-SFT models · XSTest | over-refusal | Hurt |
| Use EMD instead of KTO | Llama 7b safety tuning · four harmfulness benchmarks | harmfulness | Helped |
| Use EMD instead of NLCL | TA-SFT safety training · four harmfulness benchmarks | harmfulness | Helped |
Surprising findings
- Adding LLM-generated contrastive seemingly toxic samples reduced safety and caused non-English outputs at higher penalty weight.
- Over-refusal rises as safety improves even when the training data contains no refusal responses, suggesting refusal examples in SFT data are not the cause of over-refusal.
Limitations
- Experiments limited to model sizes up to 13B due to compute.
- Automatic safety judges (DeBERTa, OpenAI Moderation) do not guarantee every response is safe.
- Over-refusal persists and its underlying cause is not resolved.
Experimental setup
Datasets: Alpaca I-Malicious I-CoNa I-Controversial HarmfulQ XSTest AlpacaEval BoolQ PIQA OpenBookQA
Benchmarks: I-Malicious I-CoNa I-Controversial HarmfulQ XSTest AlpacaEval BoolQ PIQA OpenBookQA
Metrics: DeBERTa harmfulness score OpenAI Moderation harmful rate number of harmful responses over-refusal rate AlpacaEval preference rate multiple-choice accuracy
Baselines: SFT KTO NLCL Safety-Tuned Llamas (STL)
Builds on: ORPO Safety-Tuned Llamas KTO Alpaca
Closest papers by meaning
- Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets (ICML 2025 · similarity 0.90)
- Learning and Forgetting Unsafe Examples in Large Language Models (ICML 2024 · similarity 0.89)
- Safe LoRA: The Silver Lining of Reducing Safety Risks when Finetuning Large Language Models (NeurIPS 2024 · similarity 0.89)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.89)
- Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models (ICLR 2025 · similarity 0.89)
- Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts? (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.