Research map / Large language models / Instruction and fine-tuning

Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Safety alignment after fine-tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Expected Harm (severity × execution likelihood, modeled via execution cost) to evaluate LLM safety, finds models are inversely calibrated to cost, and uses cost-based decomposition to boost jailbreak ASR.

Problem

Current LLM safety evaluations rely on severity-based taxonomies and ASR, implicitly treating all non-refusals as equally dangerous and ignoring how feasible it is for a real actor to execute the model's response.

Approach

Defines Expected Harm as severity times execution likelihood (a decreasing function of execution cost on a 0-5 scale), reports an 'inverse risk calibration' where models refuse high-cost threats but comply with low-cost ones, and introduces cost-based decomposition of high-cost queries into low-cost sub-tasks to bypass safety filters.

Key results

• Cost + monitor decomposition raises LLaMA3.2-3B AdvBench ASR for GCG from 0.48 to 0.74, roughly 2x the attack's baseline ASR across attacks.
• Fine-tuned Qwen3-8B usefulness judge reaches 86.50% F1, close to gpt-oss-120b majority vote at 85.71%.
• Cost+monitor decomposition achieves 91% subtask bypass on Qwen3Guard-Gen-8B and 96% on Llama-Guard-3-8B, versus 2% and 5% for original prompts.

Surprising findings

• Framing decomposition as a safety-oriented rephrasing task raises gpt-4o-mini decomposition success from 19% to 96%.
• Linear probe refusal signal is bimodal across cost levels (high at cost 1 and 5, near baseline at cost 2-3), not monotonic.

Limitations

• Cost and severity labels are produced by an LLM (gpt-oss-120b) and only partially validated by human annotation on a small sample.
• Main experiments use only a 3B open-weight model plus a 20B/120B model in appendix; generality to other models is limited.
• Ethical concerns: decomposition attack could be misused, and the paper's claim that decomposition reduces perceived harm is only partially substantiated.

Experimental setup

Datasets: LMSYS-Chat-1M WildChat-4.8M AdvBench HarmBench SorryBench StrongREJECT

Benchmarks: AdvBench HarmBench SorryBench StrongREJECT

Metrics: Attack Success Rate Usefulness F1 mean severity mean cost linear probe accuracy subtask bypass rate

Baselines: Monitoring Decomposition Unrestricted Decomposition GCG AutoDAN Roleplay Many-Shot

Builds on: GCG AutoDAN Sorry-Bench HarmBench StrongREJECT

Closest papers by meaning

• STAIR: Improving Safety Alignment with Introspective Reasoning (ICML 2025 · similarity 0.90)
• WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (NeurIPS 2024 · similarity 0.90)
• The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions (ICML 2025 · similarity 0.90)
• Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy (CVPR 2025 · similarity 0.89)
• Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions (ICML 2025 · similarity 0.89)
• LLMs Encode Harmfulness and Refusal Separately (NeurIPS 2025 · similarity 0.89)