Research map / Large language models / Alignment and preferences
POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that overgenerating instruction-tuning data with stronger teachers (GPT-4o) improves safety and reduces overrefusal in student LLMs, and proposes POROver, a DPO-based post-training step that further reduces overrefusal while keeping safety.
Problem
Instruction-finetuned LLMs must balance safety against overrefusal of benign prompts that look toxic; the effect of using newer, stronger teacher models for completions on this trade-off is unexplored, and highly safe models with high overrefusal lack a targeted post-training fix.
Approach
Overgenerate eight GPT-4o completions per prompt for general and toxic instructions, rejection-sample them with ArmoRM and Llama Guard 2, and then apply DPO with preference pairs that contrast overrefused refusals (losing) with compliant helpful completions (winning) on seemingly toxic prompts, plus safe-vs-unsafe pairs on toxic prompts.
Key results
- Overgenerating general-purpose completions with GPT-4o raises the OR-Bench F1 (safety/usefulness) from 74.39% (GPT-3 data) to 91.78% (ArmoRM helpful) for Llama-3.1-8B.
- Toxic-prompt completions from GPT-4o with 20,000 added safety examples raise OR-Bench Not-Overrefusal Rate from 11.1% to 57.6% at comparable safety for Llama-3.1-8B.
- POROver (DPO) raises OR-Bench Not-Overrefusal Rate from 57.6% to 82.1% with Not-Unsafe Rate 97.9%, and AlpacaEval win rate is unchanged.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace original GPT-3 general-purpose completions with overgenerated GPT-4o completions | Llama-3.1-8B instruction finetuning · OR-Bench and XSTest | F1-Score: 74.39 -> 90.76 (OR-Bench); 90.75 -> 95.44 (XSTest) | Helped |
| Use ArmoRM helpfulness selection rather than random selection for GPT-4o completions (ablation) | Llama-3.1-8B general-purpose instruction finetuning · OR-Bench | F1-Score: 90.76 -> 91.78 | Helped |
| Use ArmoRM helpfulness selection rather than random selection for GPT-4o completions (ablation) | Llama-3.1-8B general-purpose instruction finetuning · AlpacaEval | Win Rate: 36.57 -> 39.32 | Helped |
| Train with 20,000 GPT-4o toxic-prompt completions instead of 2,000 GPT-3.5 completions | Llama-3.1-8B safety instruction finetuning · OR-Bench Seemingly Toxic | Not-Overrefusal Rate: 11.1% -> 57.6% | Helped |
| Increase GPT-4o toxic-prompt safety data from 2,000 to 20,000 examples (ablation) | Llama-3.1-8B safety instruction finetuning · OR-Bench Toxic | Not-Unsafe Rate | Helped |
| Apply DPO-based POROver using preference data for seemingly toxic and genuinely toxic prompts | Llama-3.1-8B finetuned with 20,000 GPT-4o toxic examples · OR-Bench Seemingly Toxic | Not-Overrefusal Rate: 57.6% -> 82.1% | Helped |
| Apply DPO-based POROver | Llama-3.1-8B finetuned with 20,000 GPT-4o toxic examples · OR-Bench Toxic | Not-Unsafe Rate: 98.6% -> 97.9% | Hurt |
| Apply DPO-based POROver | Llama-3.1-8B finetuned with 20,000 GPT-4o toxic examples · XSTest Seemingly Toxic | Not-Overrefusal Rate: 89.2% -> 90.8% | Helped |
| Apply DPO-based POROver | Llama-3.1-8B finetuned with 20,000 GPT-4o toxic examples · AlpacaEval | Win Rate: 38.93% (1.66 standard error) before and after | No clear effect |
| Exclude toxic prompts from preference training by setting tau = 0 (ablation) | POROver · Validation safety and usefulness | Safety and usefulness | Mixed |
| Set the containment threshold tau = 0.5 (ablation) | POROver · Validation safety and usefulness | Safety and usefulness | Mixed |
| Increase added safety data to higher amounts (ablation) | Llama-3.1-8B instruction finetuning · OR-Bench Toxic | Not-Unsafe Rate | Helped |
Surprising findings
- Models trained on GPT-4o completions for toxic prompts require about 10x more safety data (20,000 vs 2,000) to match the Not-Unsafe Rate of GPT-3.5-trained models.
- Tuning the containment threshold tau to 0 (no toxic prompts in the preference set) causes a significant safety drop.
Limitations
- Only DPO was tested as the preference optimization method.
- Relies on GPT-4o as teacher, a proprietary model, and on auto-annotators whose agreement with humans differs by 1-2%.
- Benefits may diminish as student size approaches teacher size; scaling behavior not quantified.
Experimental setup
Datasets: Alpaca Anthropic Red-teaming OR-Bench 80k AlpacaFarm
Benchmarks: AlpacaEval 2.0 OR-Bench Toxic XSTest Toxic I-CoNa Q-Harm I-Controversial OR-Bench Seemingly Toxic XSTest Seemingly Toxic JailBreakBench
Metrics: Not-Unsafe Rate Not-Overrefusal Rate F1 score AlpacaEval win rate attack success rate
Baselines: GPT-3.5 completions (original safety data) GPT-3 Alpaca completions DeBERTa reward model rejection sampling Random selection
Builds on: DPO Alpaca OR-Bench Llama Guard 2 ArmoRM
Closest papers by meaning
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.90)
- Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack (ICML 2025 · similarity 0.89)
- Learning and Forgetting Unsafe Examples in Large Language Models (ICML 2024 · similarity 0.89)
- Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models (ICLR 2025 · similarity 0.89)
- Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation (ICLR 2025 · similarity 0.89)
- SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.