Research map / Large language models / Alignment and preferences

Mission Impossible: A Statistical Perspective on Jailbreaking LLMs

NeurIPS 2024 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Gives a statistical (PAC-Bayes and simplex-volume) account of why pretrained LLMs reproduce harmful content and why jailbreaking after alignment is unavoidable, then proposes E-RLHF/E-DPO, which swaps in a safety-transformed prompt as the KL reference to expand the safe zone.

Problem

LLMs trained on uncurated text reproduce harmful behaviour, and preference-aligned models can still be jailbroken; the paper seeks a principled statistical explanation and a safer alignment objective.

Approach

Decompose prompts into query and concept, model LMs as generators of text fragments, prove a PAC-Bayesian bound implying harmful pretraining content is mimicked, and show via a simplex-volume argument that jailbreaks are unavoidable; then modify the RLHF/DPO KL reference to use a safety-transformed prompt (prefix) for harmful prompts only.

Key results

• E-DPO lowers average HarmBench ASR across adversaries from 42.00 (DPO) to 36.95, and AdvBench average ASR from 28.40 (DPO) to 20.89.
• On MT-Bench, DPO scores 6.8 and E-DPO 6.6, both above the SFT model's 6.3, so helpfulness is not degraded.
• Ablation finds applying the safe prefix to all prompts (not only harmful ones) degrades safety.

Measured effects reported in the paper

Surprising findings

• Applying the safe prefix to non-harmful prompts degrades safety, so the switch must be restricted to harmful prompts.

Limitations

• The theoretical results rely on strong assumptions (concept-determined support, semantically meaningful outputs) that are not empirically verified.
• Harmful fraction alpha is only loosely estimated; the analysis uses a fixed world mapping and does not cover multi-turn jailbreaks.
• Evaluation is limited to DPO-based experiments with one SFT model and a 100-prompt AdvBench subset.

Experimental setup

Datasets: alignment-handbook preference dataset (public) HarmBench AdvBench harmful behaviors (first 100 prompts) MT-Bench

Benchmarks: HarmBench AdvBench MT-Bench

Metrics: attack success rate (ASR) MT-Bench score

Baselines: DPO SFT model (p_SFT) RLHF

Builds on: DPO RLHF PAC-Bayesian bounds context distillation

Closest papers by meaning

• PARDEN, Can You Repeat That? Defending against Jailbreaks via Repetition (ICML 2024 · similarity 0.90)
• Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy (CVPR 2025 · similarity 0.90)
• Improving LLM Safety Alignment with Dual-Objective Optimization (ICML 2025 · similarity 0.90)
• Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (ICLR 2024 · similarity 0.90)
• Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.89)
• Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMs (ICML 2025 · similarity 0.89)