Research map / Large language models / Instruction and fine-tuning
BackdoorAlign: Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Backdoor Enhanced Safety Alignment, which prefixes a random secret prompt to a few safety examples in the fine-tuning data so the secret prompt acts as a backdoor trigger that restores safety after fine-tuning-based jailbreak attacks.
Problem
Fine-tuning aligned LLMs on a few harmful examples (FJAttack) compromises safety in LMaaS settings, and existing defenses that mix in safety examples need many of them to be effective.
Approach
Treat the limited safety examples as poisoned data and a secret random-token prompt prepended to their system prompts as a backdoor trigger, so fine-tuning binds the trigger to safe responses; the provider prepends the secret at inference to activate safety.
Key results
- On Llama-2-7B-Chat, the method reduces ASR from 34.91% (Baseline) to 3.64%, with ARC-Challenge Acc 51.88%, close to the unattacked model's 3.27% ASR.
- On GPT-3.5-Turbo, Harmfulness Score drops from 4.55 (Baseline) to 1.73 and ASR from 60.00% to 14.91%.
- Removing the secret prompt at inference raises Llama-2 ASR from 3.64% to 26.18%, indicating the trigger is effective.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add prefixed safety examples with a secret prompt and use it at inference | Llama-2-7B-Chat · Policy-Oriented Safety Evaluation Benchmarks | ASR (%): 34.91 -> 3.64 | Helped |
| Add prefixed safety examples with a secret prompt and use it at inference | GPT-3.5-Turbo · Policy-Oriented Safety Evaluation Benchmarks | ASR (%): 60.00 -> 14.91 | Helped |
| Add prefixed safety examples with a secret prompt and use it at inference | Llama-2-7B-Chat · ARC-Challenge | Accuracy (%): 50.68 -> 51.88 | No clear effect |
| Add prefixed safety examples with a secret prompt and use it at inference | GPT-3.5-Turbo · MMLU | Accuracy (%): 66.51 -> 66.37 | No clear effect |
| Select the 11 safety examples category-wise across harmful categories rather than randomly (ablation) | Backdoor Enhanced Safety Alignment · Policy-Oriented Safety Evaluation Benchmarks | ASR (%): 7.64 -> 3.64 | Helped |
| Select safety examples category-wise rather than using LLM-generated examples (ablation) | Backdoor Enhanced Safety Alignment · Policy-Oriented Safety Evaluation Benchmarks | ASR (%): 20.73 -> 3.64 | Helped |
| Add the prefixed secret prompt during inference (ablation) | Backdoor Enhanced Safety Alignment · Policy-Oriented Safety Evaluation Benchmarks | ASR (%): 26.18 -> 3.64 | Helped |
| Add an inference secret prompt to the Baseline defense without secret-prompt fine-tuning (ablation) | Baseline · Policy-Oriented Safety Evaluation Benchmarks | ASR (%): 34.91 -> 33.09 | No clear effect |
| Use 150 random tokens instead of the Llama 2 default system prompt as the secret prompt (ablation) | Backdoor Enhanced Safety Alignment · Policy-Oriented Safety Evaluation Benchmarks | ASR (%): 7.64 -> 3.64 | Helped |
| Use 150 random tokens instead of the GPT-4-generated prompt as the secret prompt (ablation) | Backdoor Enhanced Safety Alignment · Policy-Oriented Safety Evaluation Benchmarks | ASR (%): 7.27 -> 3.64 | Helped |
| Use LoRA fine-tuning (ablation) | Backdoor Enhanced Safety Alignment · Policy-Oriented Safety Evaluation Benchmarks | ASR (%): 40.36 -> 2.91 | Helped |
| Apply Backdoor Enhanced Safety Alignment in dialog-summary fine-tuning | Llama-2-7B-Chat · Dialog Summary | ASR (%): 22.55 -> 10.55 | Helped |
Surprising findings
- Random secret tokens outperform semantically meaningful system prompts (Llama 2 default, GPT-4 generated) as backdoor triggers, with lower ASR.
- Prepending the secret prompt at inference without the defense fine-tuning does not improve safety (Baseline ASR 33.09% vs 34.91%).
Limitations
- Still requires a small set of safety examples, adding a tiny fine-tuning cost.
- Only studied in the fine-tuning stage; extension to instruction tuning or RLHF safety alignment is unexplored.
Experimental setup
Datasets: pure_bad harmful examples (100) Policy-Oriented Safety Evaluation Benchmarks SAMSum sql-create-context Stanford Alpaca
Benchmarks: Policy-Oriented Safety Evaluation Benchmarks ARC-Challenge MMLU MT-Bench SAMSum sql-create-context
Metrics: Harmfulness Score Attack Success Rate ARC-Challenge Acc MMLU Acc MT-Bench Score Rouge-1 F1
Baselines: No Defense Baseline mixed safety examples fine-tuning
Builds on: Harmful example demonstration attack (FJAttack) backdoor attacks
Closest papers by meaning
- Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy (CVPR 2025 · similarity 0.91)
- h4rm3l: A Language for Composable Jailbreak Attack Synthesis (ICLR 2025 · similarity 0.90)
- STAIR: Improving Safety Alignment with Introspective Reasoning (ICML 2025 · similarity 0.90)
- Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions (ICML 2025 · similarity 0.90)
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.90)
- Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.