Research map / Large language models / Alignment and preferences
LLM Safety Alignment is Divergence Estimation in Disguise
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that popular alignment losses (KTO, BCO, DPO) act as divergence estimators between aligned and unaligned distributions, proposes KLDO and a general f-divergence optimizer family, and links resulting latent safety separation (Bhattacharyya distance) to robustness.
Problem
Why do safe and harmful prompts become separated in the hidden representation space after LLM safety alignment, and is this separation a fundamental consequence of alignment methods?
Approach
Frames alignment losses (RLHF/DPO, KTO, BCO) as variational divergence estimators between aligned and unaligned response distributions, derives alignment-consistent optimal policies, and proposes KLDO (KL-based DV objective) and FDO (general f-divergence family); compliance-refusal data is argued to yield stronger separation than preference data.
Key results
- KLDO achieves the highest overall robustness score and best average rank across the four evaluated models (e.g., overall robust score 95.02 on Llama-3.2-1B and 92.04 on Qwen2.5-1.5B).
- Bhattacharyya distance correlates negatively with ASR (SALAD r=-0.82, p<0.001) and positively with ToxiGen accuracy (r=0.66) and overall robustness (r=0.70) across methods, after model normalization.
- KLDO attains the largest separation on Gemma-2-2B (D_B=10.13), while BCO has the largest on Qwen2.5-1.5B (11.77) and Mistral (8.94); DPO yields the lowest separation in most models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KLDO instead of the strongest listed baseline for prompt safety separation | Llama3.2-1B · Safety prompt separation | Bhattacharyya Distance (D_B): 6.50 vs 5.75 | Mixed |
| Use KLDO instead of BCO | Llama2-7B · Overall safety robustness | Overall Robust Score: 80.54 -> 90.44 | Helped |
| Use KLDO instead of BCO | Gemma2-2B · Safety prompt separation | Bhattacharyya Distance (D_B): 2.91 -> 10.13 | Helped |
| Use KLDO instead of BCO | Mistral-7B-v0.1 · Overall safety robustness | Overall Robust Score: 96.29 -> 87.87* | Hurt |
| Use KLDO instead of BCO | Qwen2.5-1.5B · Overall safety robustness | Overall Robust Score: 76.01 -> 92.04 | Helped |
| Use compliance–refusal data rather than preference data | Llama3.2-1B aligned model · Safe versus harmful prompt separation and AdvBench | Bhattacharyya Distance (D_B) | Helped |
| Increase prompt representation separation | Aligned models across methods · AdvBench Clean | Pearson correlation with D_B: -0.50 (0.024) | Helped |
| Increase prompt representation separation | Aligned models across methods · AdvBench GCG | Pearson correlation with D_B: -0.50 (0.023) | Helped |
| Increase prompt representation separation | Aligned models across methods · SALAD | Pearson correlation with D_B: -0.82 (<0.001) | Helped |
| Increase prompt representation separation | Aligned models across methods · ToxiGen | Pearson correlation with D_B: 0.66 (0.0014) | Helped |
| Increase prompt representation separation | Aligned models across methods · Overall robustness | Pearson correlation with D_B: 0.70 (0.0006) | Helped |
Surprising findings
- DPO-induced divergence saturates early and yields weak latent separation, leading to lower robustness than KTO and BCO in several models.
- Preference data reduces separation and increases ASR compared with compliance-refusal data for Llama-3.2-1B.
Limitations
- Experiments limited to models up to 7B parameters due to resource constraints.
- Theory for DPO alignment consistency is only heuristic (no closed-form divergence).
- Separation metric assumes Gaussian fits of first two PCA components.
Experimental setup
Datasets: SafeAligner Alpaca-GPT4-Data AdvBench SALAD ToxiGen AlpacaEval MMLU GSM8K
Benchmarks: AdvBench SALAD ToxiGen AlpacaEval MMLU GSM8K
Metrics: Bhattacharyya distance attack success rate ToxiGen accuracy win rate exact match Pearson correlation
Baselines: DPO KTO BCO
Builds on: DPO KTO BCO MINE Donsker-Varadhan representation
Closest papers by meaning
- Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates (NeurIPS 2024 · similarity 0.89)
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.89)
- Improving LLM Safety Alignment with Dual-Objective Optimization (ICML 2025 · similarity 0.88)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.88)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.88)
- SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.