Research map / Large language models / Instruction and fine-tuning

Safety Alignment Should be Made More Than Just a Few Tokens Deep

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Safety alignment after fine-tuning · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Argues that LLM safety alignment is "shallow" (concentrated in the first few output tokens), shows this explains several known vulnerabilities, and proposes a safety-recovery data augmentation and a token-wise constrained fine-tuning objective that improve robustness on Llama-2-7B-Chat and Gemma-1.1-7B-IT.

Problem

Current safety alignment of LLMs is vulnerable to prefilling, adversarial suffix, decoding-parameter, and fine-tuning attacks; the paper asks whether a shared underlying cause is that alignment only changes the first few output tokens.

Approach

Alignment takes a shortcut by adapting only the initial refusal prefix, so harmful continuations stay largely unchanged deeper in the response; the paper counters with safety-recovery examples (harmful prefix followed by refusal) and a token-wise constrained SFT loss that limits deviation on early tokens.

Key results

• Augmented Llama-2-7B-Chat reduces prefilling attack ASR (5 tokens) from 42.1% to 2.8% and GCG ASR on HEx-PHI from 36.5% to 18.4%.
• Constrained SFT lowers Llama-2-7B-Chat harmful-examples fine-tuning ASR from 88.9% (standard SFT) to 4.6%, with comparable benign utility (e.g., GSM8K 37.4 vs 41.7).
• Augmented model AlpacaEval winrate 49.5% vs 51.8% for the initial Llama-2-7B-Chat.

Measured effects reported in the paper

Surprising findings

• Simply prefilling a refusal prefix makes unaligned base models nearly as safe as aligned models (harmfulness drops from 68.6% to 0-16% across prefixes).
• Benign fine-tuning can raise ASR early in training; ASR of SQL-fine-tuned Llama-2 reached 13.6% after just 10 steps, suggesting early-token gradients drive safety regression.

Limitations

• Augmentation is applied as post-hoc fine-tuning on already aligned Llama-2-7B-Chat rather than end-to-end alignment, since the original pipeline is unavailable.
• The constrained objective is not a perfect defense and may be vulnerable to adaptive attacks; some configurations trade off utility or safety (uniform beta ablations).
• Results rely on GPT-4 judge and HEx-PHI-based evaluation with a limited set of attacks and two model families.

Experimental setup

Datasets: HEx-PHI AdvBench MaliciousInstruct Alpaca Samsum SQL Create Context GSM8K Harmful HEx-PHI

Benchmarks: HEx-PHI AdvBench MaliciousInstruct AlpacaEval Samsum SQL Create Context GSM8K

Metrics: attack success rate harmfulness rate winrate ROUGE-1 accuracy per-token KL divergence

Baselines: Standard SFT Llama-2-7B-Chat initial model GCG attack decoding parameters exploit

Builds on: DPO KTO GCG adversarial suffix attack Superficial Alignment Hypothesis fine-tuning attacks (harmful examples)

Closest papers by meaning

• Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.94)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.93)
• Safety Layers in Aligned Large Language Models: The Key to LLM Security (ICLR 2025 · similarity 0.93)
• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.93)
• Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories (NeurIPS 2026 · similarity 0.92)
• Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.92)