Research map / Adversarial robustness and security / Attacks on text and graphs
Safety Alignment Can Be Not Superficial With Explicit Safety Signals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a dedicated [CLS] safety classification token trained alongside SFT/pretraining, and uses its predictions via attention rules and a decoding strategy with refusal guiding tokens to make LLMs robust to jailbreaks and decoding attacks.
Problem
Safety alignment of LLMs is superficial: the implicit safety decision is diluted by competing objectives, leaving models vulnerable to prefill, nested, GCG, and decoding-parameter attacks.
Approach
Introduce an explicit [CLS] token trained with a binary safe/unsafe classification loss alongside the LM/SFT loss, then use its hidden state through a strategic attention mask and re-classify at selected generation steps, with a decoding rule that inserts refusal guiding tokens when classification flips to malicious.
Key results
- Llama2-7B-CLS reduces Prefill ASR on AdvBench from 92.7% (SFT) and 39.62% (Chat) to 0.4%.
- Llama2-7B-CLS achieves 0% ASR on GCG (HarmBench) and MaliciousInstruct decoding attacks, versus 41% and 98% for SFT.
- Annealing re-classification keeps overhead under 0.2x while performing comparably to re-classifying at every step.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add explicit safety classification with strategic attention and strategic decoding | Llama2-7B · AdvBench Prefill | ASR (%): 92.7%±2.69% -> 0.4%±0% | Helped |
| Add explicit safety classification with strategic attention and strategic decoding | Llama2-7B · HarmBench GCG | ASR (%): 41.0%±2.0% -> 0.0%±0% | Helped |
| Add explicit safety classification with strategic attention and strategic decoding | Llama2-7B · AdvBench DeepInception | ASR (%): 0%±0% -> 2.0%±0% | Mixed |
| Add explicit safety classification and dynamic safety mechanisms | Mistral-7B-Instruct-v0.2 · AdvBench Prefill | ASR (%): 92.12% -> 0.4% | Helped |
| Add explicit safety classification and dynamic safety mechanisms | Mistral-7B-Instruct-v0.2 · GSM8K | score: 41.09 -> 41.77 | Helped |
| Remove the pretraining phase (ablation) | Llama2-7B-CLS · safety attack evaluation | ASR | Hurt |
| Remove strategic attention (ablation) | Llama2-7B-CLS · safety attack evaluation | ASR | Hurt |
| Disable strategic decoding and guiding-token insertion (ablation) | Llama2-7B-CLS · safety attack evaluation | ASR | Hurt |
| Use Annealing re-classification instead of Every (ablation) | Llama2-7B-CLS · safety attack evaluation | ASR: under 0.2x additional cost | No clear effect |
| Enable attention to the [CLS] token (ablation) | Llama2-7B-CLS without strategic decoding · injected adversarial tokens at generation step 10 | safety response | Helped |
| Increase the weight of [CLS] attention (ablation) | Llama2-7B-CLS without strategic decoding · injected adversarial tokens at generation step 10 | safety response | Helped |
Surprising findings
- Pretraining with the classification task gave only limited gains, attributed to noisy Llama3-Guard weak labels.
- Keyword-based safety judges are unreliable; Llama3-Guard produced false positives that GPT-4 re-evaluation was used to correct.
Limitations
- Experiments limited to models up to 10B parameters due to compute; GCG results against the SOTA augmentation baseline were not reported.
- Evaluated only on text-based attacks; multimodal extension left to future work.
- Pretraining phase gains limited by noisy weak labels from Llama3-Guard.
Experimental setup
Datasets: Wikipedia Lima Alpaca Alert Alert-DPO AdvBench HEx-PHI HarmBench PAP DeepInception MaliciousInstruct
Benchmarks: AdvBench HEx-PHI HarmBench Alert-Adversarial PAP DeepInception MaliciousInstruct MT-Bench GSM8K MMLU
Metrics: attack success rate (ASR) MT-Bench score GSM8K accuracy MMLU accuracy
Baselines: Llama2-7B-Chat Llama2-7B-SFT Llama2-7B-SFT-DPO Llama2-7B-Chat-Aug Mistral-7B-Instruct-v0.2
Builds on: BERT Superficial Safety Alignment Hypothesis DPO chain-of-thought
Closest papers by meaning
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.94)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.93)
- Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons (NeurIPS 2025 · similarity 0.92)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.92)
- Learning Safety Constraints for Large Language Models (ICML 2025 · similarity 0.91)
- Safety Layers in Aligned Large Language Models: The Key to LLM Security (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.