Research map / Adversarial robustness and security / Attacks on text and graphs

Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?

ICLR 2025 · Adversarial robustness and security › Attacks on text and graphs · approach: Adversarial prompts for language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ReG-QA, which uses an unaligned LLM to generate toxic answers to a seed question and then GPT-4o to generate natural questions that would elicit those answers, showing these natural prompts jailbreak aligned LLMs and resist defenses.

Problem

Whether safety fine-tuned LLMs generalize to natural, in-distribution prompts semantically related to toxic seed prompts, rather than only to adversarial jailbreaks.

Approach

Generate toxic answers from a seed question with an unaligned LLM (Q to A), then prompt a safety-aligned API model to produce questions that would yield those answers (A to Q), without any jailbreak objective or target-model optimization; this exploits an asymmetry between forward and reverse safety training.

Key results

• ReG-QA achieves 82% overall ASR on GPT-4-0125-preview and 93% on GPT-3.5-Turbo-1106 versus 41% and 66% for the Para-QA baseline (temperature 1).
• Against defenses on JailbreakBench with GPT-3.5, ReG-QA keeps 82% ASR under Smooth-LLM and 84% under Synonym Substitution, while PAIR and GCG drop to 0-21%.
• On GPT-4o, Gemma2-9B-IT and Qwen2.5-72B-IT at temperature 0, ReG-QA reaches 89-91% ASR@1/1k while seed questions alone give 0% ASR.

Measured effects reported in the paper

Surprising findings

• Aligned models such as GPT-4o can be prompted to generate jailbreak questions from toxic answers without denial, indicating forward safety training does not generalize to reverse (A to Q) direction.
• Naive paraphrasing of toxic questions already jailbreaks aligned models such as GPT-4 at non-trivial rates.

Limitations

• Relies on the judge GPT-4o-mini whose agreement with human labels is only about 85%.
• Attack cost is about 1000 queries per seed, far above the top leaderboard methods in some comparisons.
• Evaluated mainly on JailbreakBench's 100 seed behaviors and a single target per setting, limiting generality.

Experimental setup

Datasets: JailbreakBench

Benchmarks: JailbreakBench

Metrics: attack success rate ASR@k/n jailbreaks per seed log likelihood diversity relevance

Baselines: Para-QA Prompt and Random Search PAIR GCG AdvPrompter Smooth-LLM Synonym Substitution

Builds on: PAIR JailbreakBench Smooth-LLM Gecko

Closest papers by meaning

• Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.91)
• SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types (NeurIPS 2024 · similarity 0.90)
• Look Before You Leap: Enhance Attention and Vigilance Regarding Harmful Content with GuidelineLLM (AAAI 2025 · similarity 0.90)
• Safety Layers in Aligned Large Language Models: The Key to LLM Security (ICLR 2025 · similarity 0.89)
• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.89)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.89)