Research map / Large language models / Alignment and preferences
Improving LLM Safety Alignment with Dual-Objective Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DOOR and W-DOOR, dual-objective safety alignment combining robust refusal SFT on harmful-prefix-augmented data with NPO-based unlearning, plus reward-based token weighting, to improve LLM robustness to jailbreaks over DPO.
Problem
DPO-based safety alignment remains vulnerable to jailbreaks such as prefilling, adversarial suffix, and multi-turn attacks; its loss gives poor refusal learning and weak OOD generalization.
Approach
Decompose DPO into robust refusal training (cross-entropy on safe refusals after randomly-length harmful prefixes) and targeted unlearning via NPO on harmful responses, then reweight refusal tokens by a reward derived from the ratio of an aligned policy to the reference policy.
Key results
- On Llama-3-8B, prefilling ASR drops from 0.547 (original) and 0.210 (DPO) to 0.055 (DOOR) and 0.034 (W-DOOR).
- W-DOOR on Gemma-2-2B reduces multi-turn ASR to 0.347 versus 0.446 for DPO and prefilling ASR to 0.005 versus 0.060 for DPO.
- DPO causes the largest HellaSwag drop on Llama-3-8B (0.564 vs 0.577 original) and high XSTest over-refusal (0.456), while W-DOOR keeps HellaSwag at 0.573.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace DPO with DOOR | Llama-3-8B safety alignment · Prefilling | ASR: 0.210 -> 0.055 | Helped |
| Replace DPO with W-DOOR | Llama-3-8B safety alignment · Prefilling | ASR: 0.210 -> 0.034 | Helped |
| Replace DPO with W-DOOR | Gemma-2-2B safety alignment · Multi-Turn SORRY-Bench | ASR: 0.446 -> 0.347 | Helped |
| Replace DPO with W-DOOR | Llama-3-8B safety alignment · HellaSwag | accuracy: 0.564 -> 0.573 | Helped |
| Replace DPO with DOOR | Gemma-2-2B safety alignment · HellaSwag | accuracy: 0.478 -> 0.504 | Helped |
| Add data augmentation with harmful-response prefixes (ablation) | alignment methods · Prefilling | ASR | Helped |
| Add data augmentation with harmful-response prefixes (ablation) | DPO · Prefilling | ASR | Helped |
| Use token-level weighting for refusal tokens (ablation) | DOOR · Prefilling | ASR | Helped |
| Vary token-weight function or parameters, including sigmoid and different tau values (ablation) | W-DOOR · Prefilling | ASR | No clear effect |
| Use gradient ascent to unlearn harmful responses (ablation) | safety alignment · HellaSwag and MMLU | accuracy | Hurt |
Surprising findings
- Gradient ascent-based unlearning significantly degrades general capabilities, so naive gradient ascent is not recommended for safety alignment.
- Over-refusal decreases with extended training rather than increasing, suggesting no clear refusal/over-refusal trade-off in this setting.
Limitations
- Multi-turn robustness gains are marginal and the method lacks multi-turn or long-context training data.
- Token weighting can induce over-refusal when augmentation prefixes are randomly chosen, causing incorrect transitions to safe tokens.
- Small training set (about 400 safety pairs) and evaluation limited to two models.
Experimental setup
Datasets: SORRY-Bench HEx-PHI Alpaca HarmBench XSTest MMLU HellaSwag
Benchmarks: SORRY-Bench Multi-Turn SORRY-Bench HarmBench XSTest MMLU HellaSwag
Metrics: attack success rate accuracy over-refusal rate
Baselines: SFT DPO NPO RR TAR
Builds on: DPO negative preference optimization representation rerouting tampering attack resistance
Closest papers by meaning
- Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.93)
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.92)
- Mission Impossible: A Statistical Perspective on Jailbreaking LLMs (NeurIPS 2024 · similarity 0.90)
- Lifelong Safety Alignment for Language Models (NeurIPS 2025 · similarity 0.90)
- Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks (NeurIPS 2024 · similarity 0.90)
- Robust LLM safeguarding via refusal feature adversarial training (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.