Research map / Large language models / Instruction and fine-tuning
ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies ChatBug, a safety vulnerability caused by chat templates, and shows format mismatch and message overflow attacks bypass alignment of eight LLMs and boost existing jailbreaks, with adversarial training as a costly mitigation.
Problem
Chat templates used in instruction tuning may weaken safety alignment, since users are not bound to follow the rigid template format, letting malicious users bypass safety refusals.
Approach
Chat templates define rigid formats that LLMs must follow but users need not; attackers can omit or alter control tokens (format mismatch) or inject a prefix into the assistant field (message overflow) to elicit harmful completions, and these also boost existing jailbreaks.
Key results
- Overflow-FS reaches 100% ASR-R on Llama-3 and 98.8% on Vicuna, while Direct Instruct gives 1.1% on Llama-3.
- GPTFuzzer plus Mismatch-∅ raises ASR-R from 9.0% to 99.2%, and ArtPrompt plus Mismatch-∅ reaches 100% ASR-R.
- Adversarial training for 20 epochs drives Vicuna's ASR near 0-6% but lowers MT-Bench from 6.28 to 6.02.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply the message overflow attack with uncensored-LLM-generated prefixes (Overflow-FS) | aligned LLMs · AdvBench | ASR-R: Direct Instruct 1.1% -> Overflow-FS 100.0% | Helped |
| combine GPTFuzzer with Mismatch-∅ | GPTFuzzer on Llama-2 · AdvBench | ASR-R: 9.0% -> 99.2% | Helped |
| combine GCG with Mismatch-∅ | GCG on Llama-2 · AdvBench | ASR-R: 41.5% -> 55.4% | Helped |
| combine GCG with Overflow-S | GCG on Llama-2 · AdvBench | ASR-R: 41.5% -> 78.7% | Helped |
| combine GPTFuzzer with Overflow-S | GPTFuzzer on Llama-2 · AdvBench | ASR-R: 9.0% -> 34.4% | Helped |
| combine ArtPrompt with Mismatch-∅ | ArtPrompt on Llama-2 · AdvBench | ASR-R: 73.1% -> 100.0% | Helped |
| add Self-Reminder | Vicuna 7B-v1.5 · AdvBench | ASR-R, Mismatch-∅: 90.6% -> 23.3% | Helped |
| add SafeDecoding | Vicuna 7B-v1.5 · AdvBench | ASR-R, Overflow-S: 98.5% -> 96.2% | No clear effect |
| fine-tune on adversarial examples for 20 epochs | Vicuna 7B-v1.5 · AdvBench | ASR-R, Mismatch-∅: 90.6% -> 0.0% | Helped |
| fine-tune on adversarial examples for 20 epochs | Vicuna 7B-v1.5 · MT-Bench | score: 6.28 -> 6.02 | Hurt |
| increase Overflow-FS prompt shots (ablation) | Overflow-FS attack · AdvBench | ASR | Helped |
| mix benign examples with adversarial training examples (ablation) | Vicuna adversarial training · AdvBench | ASR | No clear effect |
Surprising findings
- Safety alignment tied to a chat template transfers: Mismatch-C with ChatML yields only 1.3% ASR-R on Llama-3 despite using a different template.
- Vicuna shares Llama-2's vocabulary yet responds differently under mismatch attacks, so the vulnerability is attributed to the template rather than the tokenizer.
Limitations
- Mitigation evaluated only on Vicuna, with only three countermeasures tested.
- Format mismatch attacks cannot be launched against Gemini and Claude because their APIs restrict templates, so closed-model coverage is partial.
- Adversarial training carries a substantial helpfulness cost (MT-Bench drop).
Experimental setup
Datasets: AdvBench
Benchmarks: AdvBench MT-Bench
Metrics: ASR-R ASR-M MT-Bench score
Baselines: Direct Instruct Self-Reminder SafeDecoding GCG GPTFuzzer ArtPrompt
Builds on: GCG GPTFuzzer ArtPrompt SafeDecoding Self-Reminder
Closest papers by meaning
- ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents (ICLR 2026 · similarity 0.91)
- Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions (ICML 2025 · similarity 0.89)
- Multilingual Jailbreak Challenges in Large Language Models (ICLR 2024 · similarity 0.89)
- WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models (NeurIPS 2024 · similarity 0.89)
- GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (ICLR 2024 · similarity 0.88)
- Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.