Research map / Large language models / Instruction and fine-tuning

Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates

NeurIPS 2024 · Large language models › Instruction and fine-tuning · approach: Calibration and knowledge in LLM tuning · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that fine-tuning aligned chat LLMs on benign data degrades safety mainly when training and test prompt templates match, and proposes Pure Tuning, Safe Testing (PTST): fine-tune without a safety prompt, test with one.

Problem

Benign fine-tuning of aligned LLMs can erode safety alignment; the paper asks how prompt templates used during fine-tuning and inference affect safety preservation while keeping downstream helpfulness.

Approach

Create an intended discrepancy between training and test prompt templates: fine-tune without the safety system prompt but apply the Llama 2 safety prompt at inference (PTST), which keeps ASR low while retaining most helpfulness gains; also tested against safety examples mixing and early stopping.

Key results

• On DirectHarm4, fine-tuning Llama 2-7B-Chat with chat:vanilla and testing with chat:llama drops ASR from 18.08% to 1.08% versus same-template testing, with GSM8K accuracy only falling from 33.51% to 30.00%.
• For GPT-3.5 Turbo fine-tuned with chat:vanilla, switching to chat:llama at inference reduces DirectHarm4 ASR from 22.75% to 4.50%, with similar helpfulness.
• Mixing 2483 safety examples drives AdvBench ASR near 0% for same-template settings but GSM-Danger ASR still rises above 10% for same-template training, while PTST keeps it around 4-5%.

Measured effects reported in the paper

Surprising findings

• Training with the Llama safety prompt (chat:llama) and testing with the same prompt increased DirectHarm4 ASR from 0% to 18.08%, more than training with chat:vanilla without a safety prompt.
• Self-Reminder and In-context Defense at test time, when the model was fine-tuned with the same template, degraded safety more than chat:llama training-and-testing.

Limitations

• Limited hyperparameter sweeps and few random seeds due to compute and API cost; many results are single runs.
• Evaluation mostly on harmful queries the original model already refuses; jailbreak robustness only tested with GCG on one setting.
• Mechanisms behind PTST's generalization across templates are not understood.

Experimental setup

Datasets: GSM8K ChatDoctor OpenOrca Orca-Math AdvBench JailbreakBench Alpaca ARC

Benchmarks: AdvBench DirectHarm4 GSM-Danger GSM8K ChatDoctor BERTScore ARC-Easy ARC-Challenge GCG attack from JailbreakBench AlpacaEval

Metrics: attack success rate exact match accuracy BERTScore harmfulness Likert score

Baselines: Self-Reminder In-context Defense chat:llama safety prompt No FT early stopping

Builds on: Qi et al. 2024 benign fine-tuning safety degradation Llama 2-Chat context distillation Self-Reminder In-context Defense

Closest papers by meaning

• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.92)
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.90)
• Single Character Perturbations Break LLM Alignment (AAAI 2025 · similarity 0.89)
• Alignment Dynamics in LLM Fine-Tuning (NeurIPS 2026 · similarity 0.89)
• LLM Safety Alignment is Divergence Estimation in Disguise (NeurIPS 2025 · similarity 0.89)
• Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking (ICLR 2025 · similarity 0.89)