Research map / Large language models / Instruction and fine-tuning
Alignment-Weighted DPO: A principled reasoning approach to improve alignment
Mengxuan Hu, Vivek Datla, Anoop Kumar, Zihan Guan, Sheng Li, Alfy Samuel, Daben Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CoT safety data to SFT | Llama-2-7B SFT · SorryBench (20 jailbreak attacks, average) | ASR average: 25.99 (Safety SFT) -> 7.57 (CoT Safety SFT) | Improved |
| add alignment-weighted DPO after CoT Safety SFT | Llama-2-7B · SorryBench (20 jailbreak attacks, average) | ASR average: 7.57 -> 3.41 | Improved |
| use AW-DPO instead of standard DPO (ablation) | Llama-3.2-3B · SorryBench (20 jailbreak attacks, average) | ASR average: 1.04 -> 0.58 | Improved |
| use AW-DPO instead of standard DPO (ablation) | Mistral-7B-v0.3 · SorryBench ASR average / MMLU | ASR and accuracy: ASR 3.78 -> 0.91; MMLU 41.45% -> 54.70% | Improved |
| add standard DPO after CoT Safety SFT | Llama-2-7B · MMLU | accuracy: 44.14% -> 41.45% | Hurt |
| add standard DPO after CoT Safety SFT | Llama-2-7B · SorryBench (20 jailbreak attacks, average) | ASR average: 7.57 -> 9.11 | Mixed |
| vary scaling factor alpha from 0.05 to 0.5 (ablation) | AW-DPO · SorryBench (Llama-3.2-3B) | ASR average: 0.69 / 0.57 / 0.58 / 0.62 | No clear effect |
| raise learning rate from 1e-6 to 5e-6 (ablation) | AW-DPO · SorryBench (Llama-3.2-3B) | ASR average: 0.58 -> 6.93 | Hurt |
| lower learning rate to 5e-8 (ablation) | AW-DPO · SorryBench (Llama-3.2-3B) | ASR average: 0.58 -> 7.57 | Hurt |
| deactivate reasoning-critical attention heads (ablation) | Llama2-7B-Chat · Reasoning vs safety probes | accuracy / safety rate: reasoning 41.42% -> 23.83%; safety 100.00% -> 99.60% | Mixed |
| use preference data transferred from Llama2-7B (ablation) | AW-DPO · SorryBench (Llama-3.2-3B) | ASR average: 0.58 -> 1.85 | Mixed |
Closest papers by meaning
- Improving LLM Safety Alignment with Dual-Objective Optimization (ICML 2025 · similarity 0.90)
- A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity (ICML 2024 · similarity 0.90)
- The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.90)
- Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.90)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.89)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
