Research map / Large language models / Language, safety and interpretability

AdvPrefix: An Objective for Nuanced LLM Jailbreaks

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Jailbreak elicitation and risk · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AdvPrefix, a plug-and-play objective that replaces the fixed "Sure, here is" jailbreak target with model-dependent prefixes selected by prefilling ASR and initial NLL, and introduces refined nuanced judges for evaluating jailbreaks.

Problem

Prefix-forcing jailbreak objectives ("Sure, here is ...") are misspecified, yielding incomplete or unfaithful responses, and overconstrained, since rigid prefixes are unnatural for victim LLMs and hinder optimization; existing judges also overestimate nuanced jailbreak success.

Approach

Select one or more victim-specific target prefixes by combining high prefilling attack success rate (judged for completeness and faithfulness) with low initial negative log-likelihood, and optimize attack prompts to elicit any of them via a log-sum-probability objective.

Key results

• Replacing GCG's default prefixes on Llama-3 raises nuanced ASR from 14% to 80% with full prompt optimization (40-token/multi-prefix setting reaches ~70% with suffix optimization).
• On Gemma-2, GCG ASR rises from 11.2% to 42.0% (single prefix, 20-token suffix).
• The refined judge reaches 91.8% average human agreement versus 86.6% for HarmBench and 89.3% for StrongReject.

Measured effects reported in the paper

Surprising findings

• Existing judges (HarmBench, JailbreakBench) overestimate nuanced ASR by up to ~30 points, mainly by counting incomplete and unfaithful responses as successes.
• Replacing "Sure, here is" with "Here is" for Llama-3 yields lower final GCG losses, showing the original prefix is overconstrained.

Limitations

• Prefix selection requires many sampled completions and a 70B judge, which is computationally costly.
• The objective does not account for loss-landscape shape and requires white-box log-probabilities, so it cannot be applied to black-box attacks.
• Responses elicited are still less harmful than those from an uncensored LLM (preference win rates 30-50%).

Experimental setup

Datasets: AdvBench (50 selected requests)

Benchmarks: AdvBench

Metrics: attack success rate failure breakdown (refusal, incomplete, unfaithful) human agreement rate F1 preference win rate

Baselines: GCG (original "Sure, here is" objective) AutoDAN HarmBench judge JailbreakBench judge StrongReject judge

Builds on: GCG AutoDAN tree attention

Closest papers by meaning

• h4rm3l: A Language for Composable Jailbreak Attack Synthesis (ICLR 2025 · similarity 0.91)
• JailPO: A Novel Black-Box Jailbreak Framework via Preference Optimization Against Aligned LLMs (AAAI 2025 · similarity 0.91)
• One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMs (ICLR 2025 · similarity 0.91)
• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (ICLR 2024 · similarity 0.89)
• Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs (NeurIPS 2024 · similarity 0.89)
• An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks (ICML 2025 · similarity 0.89)