Research map / Reinforcement learning / RL from feedback
One-Shot Safety Alignment for Large Language Models via Optimal Dualization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reduces constrained safety alignment of LMs to unconstrained alignment by pre-optimizing a closed-form smooth convex dual function, yielding one-shot model-based (CAN-M) and preference-based (CAN-P) algorithms.
Problem
Constrained RLHF for LM safety alignment usually relies on iterative primal-dual (Lagrangian) policy optimization, which is expensive, unstable, and requires repeatedly retraining LMs as dual variables change.
Approach
Derive a closed-form dual function via Donsker-Varadhan, show it is convex, smooth and locally strongly convex, minimize it over offline data to get optimal dual variables, then solve a single unconstrained alignment with the modified reward r + <lambda*, g>, implemented via pseudo-preference DPO-style training.
Key results
- Dual optimization predicts safety improvement of CAN-M-aligned LMs well, with safety constraints nearly satisfied as expected under model-based evaluation.
- Under GPT-4-turbo evaluation, CAN-M achieves an empirically better helpfulness-safety Pareto trade-off than DPO baselines and Safe-RLHF.
- CAN-P slightly underperforms CAN-M, attributed to imperfect pre-alignment of log-ratio estimates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the dual variable λ (ablation) | Model-based CAN (CAN-M) · PKU-SafeRLHF-30K test; model- and GPT-based evaluation | safety score / safety win rate | Helped |
| Increase the dual variable λ (ablation) | Model-based CAN (CAN-M) · PKU-SafeRLHF-30K test; model- and GPT-based evaluation | helpfulness score / helpfulness win rate | Hurt |
| Use preference-based CAN instead of model-based CAN | CAN-M · GPT-based evaluation | helpfulness and safety win rates | Hurt |
| Optimize the dual variable with CAN-M | Alpaca-7b-reproduced · PKU-SafeRLHF-30K test | safety improvement prediction | Helped |
| Increase the number of prompts used for offline dual optimization (ablation) | CAN dual optimization · PKU-SafeRLHF-30K | dual-variable stability | Helped |
| Use fewer than 100 responses per prompt for offline dual optimization (ablation) | CAN dual optimization · PKU-SafeRLHF-30K | dual-variable accuracy | Hurt |
Surprising findings
- Empirically optimal dual variables are conservative (larger than ground truth) when fewer than about 100 responses per prompt are used, especially for large margins.
- The number of responses per prompt matters more than the number of prompts for dual estimation.
Limitations
- Relies on Bradley-Terry preference assumption and on accuracy of proxy reward/safety models.
- Experiments use only a single safety constraint; multi-constraint case not empirically validated.
- Dual estimator with softmax plug-in is biased; accuracy depends on number of responses per prompt.
Experimental setup
Datasets: PKU-SafeRLHF-30K Alpaca-eval
Benchmarks: PKU-SafeRLHF-30K test set Alpaca-eval helpful_base prompts
Metrics: average reward score average safety score pairwise win rate vs SFT predicted vs empirical safety improvement
Baselines: SFT DPO (safety-only) DPO (helpfulness-only) Safe-RLHF (primal-dual PPO)
Builds on: DPO Safe RLHF Donsker-Varadhan variational formula Bradley-Terry model
Closest papers by meaning
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.92)
- Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization (NeurIPS 2025 · similarity 0.92)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.91)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.91)
- Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback (NeurIPS 2025 · similarity 0.90)
- BSO: Safety Alignment Is Density Ratio Matching (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.