Research map / Large language models / Language, safety and interpretability
Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces JAMBench, a 160-question benchmark that triggers OpenAI moderation guardrails, and JAM, a jailbreak that combines GUARD-optimized prefixes with cipher-character output formatting guided by a fine-tuned toxic-bert shadow model.
Problem
Existing jailbreak benchmarks rarely include questions that trigger output-level moderation guardrails (filtered-out errors), so jailbreak effectiveness against such guardrails is under-evaluated.
Approach
Fine-tune a toxic-bert shadow model to mimic OpenAI's moderation scores, then optimize cipher characters interleaved around each word in the response format to lower harm scores, combined with GUARD jailbreak prefixes so outputs bypass both input and output filters; a string-match decoder recovers the text.
Key results
- JAM achieves an average 75.17% jailbreak success rate on JAMBench versus a baseline average of 3.78%, reported as ~19.88x higher.
- JAM's average filtered-out rate is 10.21%, versus a baseline average of 54.76%, reported as ~1/6 lower.
- Without a decoder, JAM responses do not trigger the moderation guardrail, while decoded JAM responses achieve high harmful scores (e.g., 0.863 for GPT-3.5 Hate and Fairness High).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine jailbreak prefixes, optimized cipher characters, and harmful instructions | JAM · JAMBench | jailbreak success rate: 3.78% -> 75.17% | Helped |
| combine jailbreak prefixes, optimized cipher characters, and harmful instructions | JAM · JAMBench | filtered-out rate: 54.76% -> 10.21% | Helped |
| add a jailbreak prefix (ablation) | JAM on GPT-3.5 · JAMBench / Hate and Fairness Medium | jailbreak success rate: 26% -> 83% | Helped |
| use a GUARD-generated prefix instead of a DAN 12.0 prefix (ablation) | JAM on GPT-3.5 · JAMBench / Hate and Fairness Medium | jailbreak success rate: 54% -> 83% | Helped |
| fine-tune the shadow model (ablation) | JAM on GPT-3.5 · JAMBench / Hate and Fairness Medium | jailbreak success rate: 69% -> 83% | Helped |
| fine-tune the shadow model (ablation) | JAM on GPT-3.5 · JAMBench / Hate and Fairness Medium | filtered-out rate: 12% -> 4% | Helped |
| increase cipher length from 10 to 20 tokens (ablation) | JAM on GPT-3.5 · JAMBench / Hate and Fairness Medium | jailbreak success rate: 57% -> 83% | Helped |
| increase cipher length from 20 to 40 tokens (ablation) | JAM on GPT-3.5 · JAMBench / Hate and Fairness Medium | jailbreak success rate: 83% -> 77% | Hurt |
| add the decoder that removes cipher characters (ablation) | JAM on GPT-3.5 · JAMBench / Hate and Fairness Medium | average harmful score: 0.012 -> 0.583 | Helped |
| add output-complexity-aware and LLM-based audit defenses | JAM · various models | jailbreak success rate: to 0% | Hurt |
Surprising findings
- Input-level harmful scores of existing benchmark questions often fail to trigger moderation flags, and ICA achieves 0% success on GPT models.
- Removing jailbreak prefixes sharply reduces success rate, showing prefixes are necessary alongside cipher characters.
Limitations
- Relies on a shadow model fit to OpenAI moderation API scores, with transferability to other guardrails only partially shown.
- Llama-3 evaluated via HuggingChat black-box interface, and Llama-3 filtered-out rates not reported.
- Evaluation uses GPT-3.5 as judge and a single GPU; small benchmark of 160 questions.
Experimental setup
Datasets: Toxic Comment Classification Challenge In-the-Wild question set HarmBench JailbreakBench AdvBench
Benchmarks: JAMBench In-the-Wild question set HarmBench JailbreakBench
Metrics: jailbreak success rate filtered-out rate perplexity score average harmful score
Baselines: GCG ICA PAIR CipherChat GUARD
Builds on: GUARD GCG CipherChat PAIR
Closest papers by meaning
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.90)
- Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs (NeurIPS 2024 · similarity 0.90)
- h4rm3l: A Language for Composable Jailbreak Attack Synthesis (ICLR 2025 · similarity 0.89)
- Multi-Turn Jailbreaking Large Language Models via Attention Shifting (AAAI 2025 · similarity 0.89)
- An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks (ICML 2025 · similarity 0.89)
- Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.