Research map / Large language models / Instruction and fine-tuning

Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Safety alignment after fine-tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that random post-fine-tuning weight noise can undo harmful fine-tuning but hurts utility, and proposes Panacea, which optimizes an adaptive perturbation applied after fine-tuning via max-max optimization to reduce harmful score while preserving fine-tuning accuracy.

Problem

Harmful fine-tuning attacks in fine-tuning-as-a-service compromise safety alignment; existing pre-fine-tuning defenses (Vaccine, RepNoise) are fragile as fine-tuning proceeds, and naive random perturbations recover safety only at the cost of downstream accuracy.

Approach

Optimize, during fine-tuning, an adaptive norm-bounded perturbation that maximizes the increase in harmful loss when added to the model, and apply it to the fine-tuned weights after fine-tuning, using alternating inner (closed-form gradient-direction) and outer updates.

Key results

• Panacea reduces average harmful score by 21.2% versus SFT across harmful ratios while FA rises 0.4% (Llama2-7B, GSM8K default setting).
• On aligned models, Panacea lowers average harmful score from 43.5 (SFT) to 23.7 across Llama2-7B-Chat, Gemma2-9B-It and Qwen2-7B-Instruct.
• On AdvBench at harmful ratio 0.2, Panacea's harmful score is 10.58% versus 48.65% for SFT.

Measured effects reported in the paper

Surprising findings

• Random Gaussian perturbation to the fine-tuned model reduces harmful score but substantially degrades fine-tuning accuracy, motivating the adaptive perturbation.
• Directly optimizing lambda*h(w) - g(w) fails (model outputs become meaningless at large lambda), whereas the perturbation-based formulation does not.

Limitations

• Perturbation optimization doubles fine-tuning clock time relative to SFT (0.17 h to 0.42 h), adding computation.
• Fine-tuning accuracy on clean data drops slightly and performance degrades at very large perturbation scales or rho.
• Evaluation relies on a GPT-3.5-turbo judge and a moderation model for harmfulness, with a limited set of tasks and models.

Experimental setup

Datasets: BeaverTails RepNoise alignment/harmful data GSM8K SST2 AlpacaEval AGNEWS AdvBench Sorry-Bench LLM-LAT

Benchmarks: BeaverTails test set (HS) GSM8K SST2 AlpacaEval AGNEWS AdvBench Sorry-Bench

Metrics: harmful score fine-tuning accuracy fulfillment rate

Baselines: SFT Vaccine RepNoise Booster Antidote Safe LoRA ConstrainSFT SaLoRA

Builds on: Vaccine RepNoise Booster SAM

Closest papers by meaning

• Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack (ICML 2025 · similarity 0.93)
• Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation (ICLR 2025 · similarity 0.92)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.92)
• Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models (ICLR 2025 · similarity 0.91)
• Safe LoRA: The Silver Lining of Reducing Safety Risks when Finetuning Large Language Models (NeurIPS 2024 · similarity 0.91)
• Representation Noising: A Defence Mechanism Against Harmful Finetuning (NeurIPS 2024 · similarity 0.90)