Research map / Large language models / Alignment and preferences
EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EvoRefuse, an evolutionary prompt-optimization method that maximizes an ELBO-based surrogate of LLM refusal probability to generate diverse pseudo-malicious instructions, releasing the EvoRefuse-Test benchmark (582 items) and EvoRefuse-Align alignment data (3,000 items).
Problem
LLMs over-refuse semantically harmless 'pseudo-malicious' instructions, but existing curation methods (manual writing, rewriting, gradient search) lack scalability or fail to produce diverse, reliably refusal-inducing prompts for evaluation and mitigation.
Approach
An evolutionary search with mutation (deceptive context, sensitive words, extreme emotion), recombination, ELBO-based fitness (refusal classifier probability plus length-normalized response log-probability), and simulated annealing selection to evolve harmless instructions that elicit confident refusals.
Key results
- EvoRefuse-Test yields 85.34% higher average refusal rate than the strongest prior benchmark across 9 LLMs without a safety system prompt, plus 34.86% greater lexical diversity and 40.03% higher response log-probability.
- SFT on EvoRefuse-Align reduces over-refusals of LLaMA3.1-8B-Instruct by 29.85% relative to the best fine-tuning baseline; DPO reduces them by 45.96% with a 4.82% safety decrease.
- Gradient attribution shows over-refusal is driven by shortcut reliance on sensitive keywords, with early transformer layers carrying the highest information flow.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate prompts with EvoRefuse instead of the strongest prior benchmark pipeline | Over-refusal evaluation across 9 LLMs · EvoRefuse-Test, without safety-prior system prompt | Average refusal rate: 85.34% | Helped |
| Generate prompts with EvoRefuse instead of the next-best benchmark | Over-refusal evaluation across 9 LLMs · EvoRefuse-Test, with safety-prior system prompt | Average refusal rate: 140.41% | Helped |
| Use EvoRefuse-Test rather than the second-best dataset | LLM response confidence on pseudo-malicious prompts · EvoRefuse-Test | Response Log-Prob: 40.03% | Helped |
| Use EvoRefuse-Test rather than the second-best baseline | Prompt lexical diversity · EvoRefuse-Test | Lexical diversity: 34.86% | Helped |
| Fine-tune with EvoRefuse-Align using SFT instead of the best fine-tuning baseline | LLaMA3.1-8B-Instruct · Pseudo-malicious evaluation datasets | Over-refusal rate: 29.85% | Helped |
| Fine-tune with EvoRefuse-Align using DPO instead of the best fine-tuning baseline | LLaMA3.1-8B-Instruct · Pseudo-malicious evaluation datasets | Over-refusal rate: 45.96% | Helped |
| Fine-tune with EvoRefuse-Align using DPO rather than SFT | LLaMA3.1-8B-Instruct · Pseudo-malicious evaluation datasets | Over-refusal rate: 45.96% (DPO); 29.85% (SFT) | Helped |
| Remove recombination (ablation) | EvoRefuse optimization · Prompt optimization | Convergence speed | Hurt |
| Remove fitness evaluation (ablation) | EvoRefuse optimization · Prompt optimization | Optimization stability | Hurt |
| Replace EvoRefuse optimization with the OR-Bench pipeline (ablation) | Prompt optimization · Prompt fitness progression | Optimization stability | Hurt |
| Replace EvoRefuse optimization with the PHTest pipeline (ablation) | Prompt optimization · Prompt fitness progression | Convergence speed | Hurt |
| Start optimization from unsafe instructions rather than pseudo-malicious seeds (ablation) | EvoRefuse · LLaMA3.1-8B-Instruct optimization | PRR: 75% | Helped |
Surprising findings
- Claude3.5 and other frontier models show refusal rates on EvoRefuse-Test that are still low for most models (e.g., 0.12 to 0.38 for most), suggesting the benchmark is not saturated but also that most models refuse rarely.
- Few-shot prompting reduced over-refusal on EvoRefuse-Test for LLaMA but did not help on other pseudo-malicious sets, with mixed effects across benchmarks.
Limitations
- Relies on GPT-4o for mutation, recombination, and safety judgment, so generation depends on a closed API model.
- Fitness uses a distilroberta refusal classifier as a proxy for the refusal likelihood, and the ELBO is not guaranteed to be order-preserving for the true objective.
- Entropy term is approximated as constant, and the convergence result only holds for record-best fitness.
Experimental setup
Datasets: TRIDENT-Core XSTest OR-Bench PHTest AdvBench HarmBench JailBreakV
Benchmarks: EvoRefuse-Test XSTest OKTest SGTest HITest OR-Bench PHTest AdvBench HarmBench JailBreakV
Metrics: prefix refusal rate classifier refusal rate MSTTR HDD MTLD log-probability LongPPL manual safety annotation
Baselines: XSTest OKTest SGTest HITest OR-Bench PHTest PromptAgent Few-Shot Prompting DRO
Builds on: DPO LoRA OR-Bench pipeline PHTest pipeline PromptAgent
Closest papers by meaning
- OR-Bench: An Over-Refusal Benchmark for Large Language Models (ICML 2025 · similarity 0.90)
- Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-Tuning (ICML 2025 · similarity 0.90)
- SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal (ICLR 2025 · similarity 0.90)
- Refusal Direction is Universal Across Safety-Aligned Languages (NeurIPS 2025 · similarity 0.89)
- Improving LLM Safety Alignment with Dual-Objective Optimization (ICML 2025 · similarity 0.89)
- Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.