Research map / Large language models / Alignment and preferences
Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BFPO, a supervised (DPO-like) objective that re-parameterizes multi-objective safety-helpfulness RLHF using a safety-aware labeling function, and shows improved harmlessness and helpfulness for Mistral-7B and Zephyr-7B without red-teaming-specific annotation.
Problem
Balancing safety and helpfulness when fine-tuning LLMs: naive mixing of preference datasets produces unsafe or unhelpful models, and RLHF-based safety alignment requires costly reward modeling and red teaming.
Approach
Re-parameterize the multi-objective RLHF objective (bilinear safety x helpfulness reward) into a single supervised squared loss whose label function g_I depends on the safety labels of both responses, and train on balanced batches from helpfulness and safety datasets.
Key results
- BFPO on Mistral-7B reaches average safety score Savg 77.16% vs 49.03% for DPO on the naive mix, with Alpaca helpfulness 13.33%.
- Applied to Zephyr-7b-beta with 1.5K red-teaming prompts, BFPO raises harmlessness from 60.99% to 73.90% and Alpaca from 10.99 to 14.41.
- BFPO's generative harmlessness rises from 39.71% (DPO naive mix) to 95.24%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use BFPO instead of DPO | Mistral-7B-v0.1 · Generative safety benchmark | Gen.: 39.71% to 95.24% | Helped |
| use BFPO instead of DPO-S | Mistral-7B-v0.1 · AlpacaEval 2.0 | Alpaca: 4.34 to 13.33 | Mixed |
| use BFPO instead of DPO-H (Zephyr) | Mistral-7B-v0.1 · Harmlessness benchmark | Savg.: 60.99 to 77.16 | Helped |
| fine-tune Zephyr-7b-beta with BFPO using 1.5K red teaming prompts | Zephyr-7b-beta · Generative safety benchmark | Gen.: 62.94% to 88.79% | Helped |
| set shifting parameter α to 0 (ablation) | BFPO · Generative safety benchmark | Gen.: 95.24 -> 92.87 | Hurt |
| set shifting parameter α to 0 (ablation) | BFPO · Harmlessness benchmark | Savg.: 77.16 -> 75.98 | Hurt |
| remove balanced buffer training and randomly sample from mixed datasets (ablation) | BFPO · Generative safety benchmark | Gen.: 95.24 -> 88.76 | Hurt |
| remove balanced buffer training and randomly sample from mixed datasets (ablation) | BFPO · Harmlessness benchmark | Savg.: 77.16 -> 74.45 | Hurt |
Surprising findings
- MORL with a public reward model (ArmoRM) yields only marginal safety gains, attributed to reward scores that are not sufficiently distinguishable.
Limitations
- Relies on specific label formats for helpfulness and safety, which may limit transfer to other tasks.
- Only two objectives are handled; extending to more objectives is left open.
- Safety labels for helpfulness data are assumed (all responses safe), and PairRM is used for helpfulness rankings in the red-teaming stage.
Experimental setup
Datasets: UltraChat PKU-SafeRLHF UltraFeedback CrowS-Pairs BBQ WinoGrande ETHICS Moral Permissibility Simple Ethics Questions ToxicGen BigBench HHH Alignment AdvBench
Benchmarks: AlpacaEval 2.0 CrowS-Pairs BBQ WinoGrande ETHICS Moral Permissibility Simple Ethics Questions ToxicGen BigBench HHH Alignment AdvBench Real Toxicity Prompts ALERT
Metrics: win rate (Alpaca) discriminative accuracy harmless rate (generative) average safety score
Baselines: DPO IPO MORL DPO-H (Zephyr) DPO-S Beaver Llama2
Builds on: DPO IPO multi-objective RLHF Zephyr Mistral-7B
Closest papers by meaning
- Rule Based Rewards for Language Model Safety (NeurIPS 2024 · similarity 0.90)
- Active Preference Learning for Large Language Models (ICML 2024 · similarity 0.89)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.89)
- Multi-Reference Preference Optimization for Large Language Models (AAAI 2025 · similarity 0.89)
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.89)
- MallowsPO: Fine-Tune Your LLM with Preference Dispersions (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.