Research map / Large language models / Language, safety and interpretability

Many-shot Jailbreaking

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Jailbreak elicitation and risk · 21 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that many-shot jailbreaking (prompting LLMs with hundreds of harmful demonstrations in long context) succeeds with power-law scaling in shot count across models and tasks, and that standard SL/RL mitigations shift the intercept but not the exponent.

Problem

Long-context windows open a new attack surface: whether many harmful in-context demonstrations can jailbreak aligned LLMs, how this effectiveness scales with the number of shots, and whether standard alignment and prompt-based mitigations can prevent it.

Approach

Scales few-shot jailbreaking to hundreds of shots using a helpful-only model to generate harmful demonstrations, finds that attack effectiveness (negative log-probability of harmful completion) follows a power law in number of shots, and uses this scaling-law view to evaluate mitigations by their effect on the intercept versus exponent.

Key results

• Claude 2.0 adopts harmful behaviors on malevolent personality evals with near 100% rate beyond 128 shots, and the MSJ attack on malicious use cases did not plateau at ~70,000 token strings.
• MSJ effectiveness follows power laws in negative log-probability of harmful response across models; around 128 shots suffice for all tested models (GPT-3.5, GPT-4, Claude 2.0, Llama 2 70B, Mistral 7B) to adopt harmful behavior.
• SL and RL on targeted data increase the power-law intercept but leave the exponent largely unchanged, so mitigations delay but do not prevent MSJ at longer context lengths; CWD prompt defense reduced attack success to 2% versus ICD's 61% to 54%.

Measured effects reported in the paper

Surprising findings

• Changing prompt formatting (swapped, translated, or Q&A tags) changes the intercept but not the slope, and may make the attack more effective.
• Composing MSJ with the GCG white-box suffix has mixed effects depending on shot count, while composing with a black-box competing-objectives attack increases harm at all context lengths.

Limitations

• Experiments largely on Claude 2.0 and closed models; vanilla MSJ requires API access with faux dialogue insertion.
• Mitigation analysis limited to SL and RL variants and two prompt-based defenses; the causal mechanism behind the power law is not established (only hypothesized via induction heads).

Experimental setup

Datasets: Malicious Use Cases (generated by helpful-only model) Opportunities to Insult (generated from helpfulness preference model questions) Malevolent personality evals (Perez et al., 2022) LogiQA TruthfulQA Winogrande TriviaQA CommonsenseQA CREAK HarmBench

Benchmarks: Malicious Use Cases Malevolent Personality Evals (psychopathy) Opportunities to Insult HarmBench

Metrics: % harmful responses negative log-likelihood of harmful response power-law exponent and intercept

Baselines: few-shot jailbreaking competing objectives attack GCG suffix In-Context Defense HarmBench jailbreak methods

Builds on: few-shot jailbreaking (Rao et al., 2023) in-context attack (Wei et al., 2023) competing objectives attack (Wei et al., 2023) GCG (Zou et al., 2023) in-context defense (Wei et al., 2023)

Closest papers by meaning

• BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models (NeurIPS 2025 · similarity 0.90)
• JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.89)
• MMJ-Bench: A Comprehensive Study on Jailbreak Attacks and Defenses for Vision Language Models (AAAI 2025 · similarity 0.89)
• Systematic Scaling Analysis of Jailbreak Attacks in Large Language Models (NeurIPS 2026 · similarity 0.89)
• Weak-to-Strong Jailbreaking on Large Language Models (ICML 2025 · similarity 0.89)
• The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense (NeurIPS 2025 · similarity 0.89)