Research map / Large language models / Language, safety and interpretability

Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Jailbreak elicitation and risk · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Improves few-shot jailbreaking (FSJ) for limited-context open-source LLMs by injecting target-specific special tokens into demos and applying demo-level random search over a Mistral-generated demo pool, reaching high ASRs on aligned models and defended Llama-2.

Problem

Many-shot jailbreaking needs long context, and vanilla few-shot jailbreaking is ineffective against well-aligned open-source LLMs such as Llama-2-Chat within limited context windows (≤8192), especially under defenses.

Approach

Inject the target model's conversation-template special tokens (e.g., [/INST]) into harmful demonstrations and optimize the choice of demonstrations via a demo-level random search that minimizes the loss of an affirmative first token, using a pool of harmful responses from a weakly aligned model.

Key results

• 8-shot I-FSJ with demo RS and [/INST] tokens reaches 100% rule-based / 96% LLM-based ASR on Llama-2-7B-Chat over AdvBench (100-token decoding), and 92.5% rule-based on HarmBench.
• On Llama-2-7B-Chat with defenses such as SmoothLLM (insert 20%), I-FSJ 8-shot with RS reaches 82-93% ASR, versus near 0% for 2-shot without RS.
• On GPT-4 with the </text> token, 2-shot I-FSJ with demo RS reaches 94% rule-based and 86% LLM-based ASR on AdvBench.

Measured effects reported in the paper

Surprising findings

• Injecting the correct [/INST] token is critical: using [INST] or Qwen's tokens yields near-zero ASR on Llama-2-7B-Chat.
• Vanilla 8-shot FSJ without demo-level random search achieves 0% ASR on Llama-2-7B-Chat, while the same setup with RS reaches near 100%.

Limitations

• Assumes the target's conversation template is known; closed-source models require guessing special tokens.
• Effectiveness may drop if future models obfuscate or randomize their special tokens.
• Evaluation on AdvBench uses only 50 behaviors, and the judge is Llama Guard rather than human or GPT-4 judges.

Experimental setup

Datasets: AdvBench HarmBench

Benchmarks: AdvBench HarmBench

Metrics: attack success rate (rule-based) attack success rate (LLM-based, Llama Guard)

Baselines: GCG PAIR AutoDAN PAP PRS ICA vanilla FSJ

Builds on: many-shot jailbreaking random search jailbreak SmoothLLM SafeDecoding

Closest papers by meaning

• Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs (NeurIPS 2024 · similarity 0.90)
• Weak-to-Strong Jailbreaking on Large Language Models (ICML 2025 · similarity 0.89)
• JailPO: A Novel Black-Box Jailbreak Framework via Preference Optimization Against Aligned LLMs (AAAI 2025 · similarity 0.89)
• Injecting Universal Jailbreak Backdoors into LLMs in Minutes (ICLR 2025 · similarity 0.89)
• One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMs (ICLR 2025 · similarity 0.88)
• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (ICLR 2024 · similarity 0.88)