Research map / Large language models / Alignment and preferences

BSO: Safety Alignment Is Density Ratio Matching

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Bregman Safety Optimization (BSO), a single-stage DPO-style family of losses that derives safety-aware preference optimization as Bregman density-ratio matching, recovering SafeDPO as the logistic-generator special case.

Problem

Safety-aware alignment for language models usually needs multi-stage pipelines, auxiliary reward/cost models, or online RL; existing direct preference methods add safety through heuristic modifications without a principled derivation.

Approach

Shows the likelihood ratio of the optimal safe policy decomposes into reference ratio, helpfulness preference ratio, and a safety correction, so safety alignment becomes density ratio matching; minimizing Bregman divergences with a convex generator yields single-stage losses (SBA generator chosen) that recover SafeDPO as a special case.

Key results

• BSO shifts the safety-helpfulness Pareto frontier outward on both Qwen2.5-0.5B and Llama3.2-3B relative to SFT, Safe RLHF, SACPO, and SafeDPO (model-based evaluation).
• Ablation on the safety penalty C shows a middle range (best around C=30) gives the best trade-off, with large C (>=40) degrading reward.
• On Llama3.2-3B with LLM-based judging, BSO reports the highest helpfulness and harmlessness among compared methods.

Surprising findings

• Reported drop in performance for large safety penalty C indicates over-conservatism rather than monotone safety gains.
• The paper's main text states that DPO-style logistic weighting saturates, so SafeDPO under-weights safe-winner pairs.

Limitations

• Experiments are limited to a single dataset (PKU-SafeRLHF-30K) and two small backbones.
• The theoretical guarantee assumes sufficient model capacity and full-support data; the practical algorithm relies on a data transformation (swap/drop) and a moderate C rather than the exact objective.
• Evaluation relies on beaver reward/cost models and an LLM judge rather than human evaluation.

Experimental setup

Datasets: PKU-SafeRLHF-30K

Benchmarks: PKU-SafeRLHF-30K test split

Metrics: helpfulness reward harmless ratio LLM-judged helpfulness score LLM-judged safe proportion

Baselines: SFT Safe RLHF SACPO SafeDPO

Builds on: DPO SafeDPO Safe RLHF BPO KLIEP

Closest papers by meaning

• Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization (NeurIPS 2025 · similarity 0.93)
• AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning (ICLR 2026 · similarity 0.92)
• One-Shot Safety Alignment for Large Language Models via Optimal Dualization (NeurIPS 2024 · similarity 0.90)
• Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback (NeurIPS 2025 · similarity 0.89)
• Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.89)
• Stepwise Alignment for Constrained Language Model Policy Optimization (NeurIPS 2024 · similarity 0.89)