Research map / Large language models / Instruction and fine-tuning

Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets

ICML 2025 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Safe Delta, a post-fine-tuning defense that greedily selects delta parameters by utility-safety ratio under a safety-loss budget and adds an OBS-style safety compensation vector to preserve safety across diverse fine-tuning datasets.

Problem

Fine-tuning aligned LLMs via API, with harmful or benign-but-uncleaned data, degrades safety; existing defenses (data augmentation, weight modification) fail to preserve safety across datasets of varying size and task while retaining utility gains.

Approach

Estimate per-parameter safety degradation with an OBS/Hessian-based formula computed once on the original model and a safety dataset, select delta parameters greedily by utility-to-safety-cost ratio under a layer-wise budget, and add a closed-form compensation vector to offset residual safety loss on the unselected weights.

Key results

• On PureBad fine-tuning of Llama-2-7B-Chat, Safe Delta reduces ASR from 95.76% (fine-tuned) to 3.33% while MT-Bench stays at 6.18 (vs 5.43 fine-tuned).
• On Dirty Summary, Safe Delta achieves the highest utility (ROUGE-1 F1 0.489) with ASR 5.15%, versus 63.94% for the fine-tuned model and 46.97% for SafeInstr.
• Ablation on Dirty Summary: removing the safety compensation vector raises ASR from 5.15% to 26.97%, while utility changes minimally.

Measured effects reported in the paper

Surprising findings

• Safe LoRA and Resta, despite strong safety on some benign data, reduce utility noticeably; Safe Delta avoids this trade-off.
• Content filtering with Llama-Guard-3-8B is ineffective on benign-content datasets (AOA, Math), where ASR remains high after fine-tuning.

Limitations

• Greedy weight selection may be suboptimal; a smarter selection could improve results.
• Vulnerable to attackers who craft fine-tuning data to avoid correction by the method.
• Only evaluated on text-based LLMs, not multimodal models.

Experimental setup

Datasets: PureBad Identity Shift (Alpaca-based) SAMSum GSM8K AdvBench BeaverTails

Benchmarks: MMLU MT-Bench Policy-Oriented Safety Evaluation Benchmarks SAMSum (ROUGE-1) GSM8K OR-Bench GCG AutoDAN PAIR

Metrics: attack success rate harmfulness score MMLU accuracy MT-Bench score ROUGE-1 F1 accuracy over-refusal rate

Baselines: SafeInstr BEA Safe LoRA Resta

Builds on: Optimal Brain Surgeon Safe LoRA Resta Backdoor Enhanced Alignment Model Tailor

Closest papers by meaning

• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.92)
• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.91)
• SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types (NeurIPS 2024 · similarity 0.91)
• Safe LoRA: The Silver Lining of Reducing Safety Risks when Finetuning Large Language Models (NeurIPS 2024 · similarity 0.90)
• Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMs (ICLR 2025 · similarity 0.90)
• What Makes and Breaks Safety Fine-tuning? A Mechanistic Study (NeurIPS 2024 · similarity 0.90)