Research map / Large language models / Language, safety and interpretability

Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions

ICML 2025 · Large language models › Language, safety and interpretability · approach: Jailbreak elicitation and risk · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HarmScore, a metric combining actionability and informativeness of jailbroken responses, and Speak Easy, a multi-step multilingual attack framework that substantially raises ASR and HarmScore for GPT-4o, Qwen2-72B and Llama-3.3-70B.

Problem

LLMs remain vulnerable to jailbreaks, but existing jailbreak evaluations rely on technical attacks and binary attack success rate, which may not reflect whether responses truly help non-expert users carry out harm in simple everyday interactions.

Approach

Identifies actionability and informativeness as the key attributes making jailbroken responses harmful, defines HarmScore as the geometric mean of learned actionability and informativeness scores (gated by refusal strings), and attacks via simple multi-step query decomposition plus multilingual querying with learned response selection.

Key results

• Speak Easy raises GPT-4o ASR on direct requests from 0.092 to 0.555 on average across four benchmarks, and HarmScore from 0.180 to 0.759.
• Integrating Speak Easy into GCG-T and TAP-T yields an average absolute increase of 0.313 in ASR and 0.398 in HarmScore across LLMs and benchmarks.
• HarmScore achieves Pearson correlation 0.726 with human judgment on HarmBench subsamples, versus 0.638 for HarmBench-classifier ASR and 0.723 for GPT-4o ASR.

Measured effects reported in the paper

Surprising findings

• GPT-4o refused 81% of single-step multilingual queries but only 1.5% when the query was decomposed into three subqueries.
• GCG-T used alone leads to minimal change or even a decline in attack success, but adding Speak Easy substantially improves it.

Limitations

• HarmScore struggles to assess actionability for misinformation and cybercrime queries focused on content generation, leading to lower correlation.
• Relies on GPT-4o-based refusal and labeling and translation via Azure AI Translator, which may introduce bias.
• Actionability scores are low at the subquery level, and gains saturate beyond six languages.

Experimental setup

Datasets: HarmBench AdvBench SORRY-Bench MedSafetyBench HH-RLHF Stack-Exchange-Preferences WildVis WildChat

Benchmarks: HarmBench AdvBench SORRY-Bench MedSafetyBench

Metrics: Attack Success Rate HarmScore Pearson correlation with human judgment preference accuracy

Baselines: Direct Request GCG-T TAP-T HarmBench classifier ASR GPT-4o judge ASR FsfairX-LLaMA3-RM-v0.1 ArmoRM-Llama3-8B-v0.1

Builds on: DPO GCG TAP HarmBench multilingual jailbreak

Closest papers by meaning

• h4rm3l: A Language for Composable Jailbreak Attack Synthesis (ICLR 2025 · similarity 0.92)
• Multilingual Jailbreak Challenges in Large Language Models (ICLR 2024 · similarity 0.91)
• Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMs (ICML 2025 · similarity 0.91)
• Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy (CVPR 2025 · similarity 0.91)
• An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks (ICML 2025 · similarity 0.91)
• Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs (NeurIPS 2024 · similarity 0.91)