Research map / Large language models / Language, safety and interpretability

Weak-to-Strong Jailbreaking on Large Language Models

ICML 2025 · Large language models › Language, safety and interpretability · approach: Jailbreak elicitation and risk · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes weak-to-strong jailbreaking, which steers a large safe LLM's decoding using the log-probability ratio of a small unsafe and small safe model pair, achieving over 99% attack success with one forward pass of the target.

Problem

Safety-aligned open-source LLMs can be jailbroken, but existing automated jailbreaks are computationally costly, especially for very large models such as 70B and 405B.

Approach

Shallow safety alignment means safe and unsafe models differ mainly in initial decoding tokens; a small unsafe model's logit shift (unsafe/safe ratio of 7B models) is applied at inference to a much larger safe model's next-token distribution, steering it toward harmful outputs without changing its weights or searching prompts.

Key results

• Weak-to-strong attack on Llama2-13B reaches 99.4% ASR on AdvBench and 99.0% on MaliciousInstruct; on Llama2-70B it reaches 99.2% and 100.0%.
• Using a 1.3B Sheared-LLaMA weak model achieves 74.0% ASR on AdvBench against Llama2-70B-Chat.
• Gradient ascent defense on Llama2-13B lowers ASR by 20%-40% for decoding-based attacks but only 5%-10% for weak-to-strong, with about 0.04 TruthfulQA accuracy drop.

Measured effects reported in the paper

Surprising findings

• Using the unaligned base model as the weak unsafe model still yields high ASR on 13B and 70B targets (e.g., 39.8% and 35.7% on AdvBench for the Dagger variant).
• Weak-to-strong outputs have low BLEU (below 0.3) with the weak unsafe model, suggesting the attacked model is not simply copying it.

Limitations

• Requires access to output logits and a shared vocabulary between weak and strong models, so closed-source models are not evaluated.
• White-box threat model; adds inference compute cost from loading two small models.
• Defense evaluated only with a simple gradient ascent method; weak-to-strong attack remains only partially mitigated (5%-10% ASR drop).

Experimental setup

Datasets: AdvBench MaliciousInstruct 100 adversarial examples from released fine-tuning dataset databricks-dolly-15k (OpenQA) GSM8K TruthfulQA SALAD-Bench SORRY-Bench

Benchmarks: AdvBench MaliciousInstruct SALAD-Bench SORRY-Bench GSM8K TruthfulQA

Metrics: attack success rate harm score GPT-4 score BLEU ROUGE sentence similarity accuracy

Baselines: GCG Prefix Injection SelfCipher DeepInception Adversarial Decoding Adversarial Fine-tuning

Builds on: contrastive decoding DExperts proxy-tuning GCG emulator fine-tuning

Closest papers by meaning

• Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (ICLR 2024 · similarity 0.92)
• An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks (ICML 2025 · similarity 0.92)
• One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMs (ICLR 2025 · similarity 0.91)
• JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.91)
• Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment (CVPR 2025 · similarity 0.91)
• JailPO: A Novel Black-Box Jailbreak Framework via Preference Optimization Against Aligned LLMs (AAAI 2025 · similarity 0.90)