Research map / Large language models / Language, safety and interpretability
COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts the COLD energy-based constrained decoding with Langevin dynamics to generate controllable, fluent, and stealthy jailbreak prompts (suffix, paraphrase, and position-constrained attacks) against white-box and GPT LLMs.
Problem
Existing white-box jailbreak methods like GCG yield gibberish suffixes easily caught by perplexity filters and offer no way to impose controllable attributes (sentiment, style, position, paraphrasing) on the attack.
Approach
Formulates controllable attack generation as constrained sampling and adapts COLD (energy-based Langevin dynamics over logits plus LLM-guided decoding) so that attack success, fluency, semantic similarity, and lexical/sentiment constraints are encoded as composable energy functions.
Key results
- Suffix attack with continuation constraint reaches 100% ASR and 92% ASR-G on Vicuna with PPL 32.96 versus GCG's 821.53, and 92% ASR with PPL 24.83 on Llama2.
- Paraphrase attack achieves 96-98% ASR (86% on Llama2) and 74-90% ASR-G while keeping BERTScore above 0.7.
- Position-constrained attacks reach ASR above 80% across sentiment, lexical, format, and style constraints, and COLD-Attack takes about 20 minutes per sample on a V100, roughly 10x faster than GCG.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COLD-Attack instead of AutoDAN-Zhu for continuation-constrained suffix generation | LLM jailbreak prompts · AdvBench subset, continuation constraint, evaluated LLMs | PPL: 33.43 -> 32.96; 50.47 -> 30.55; 79.53 -> 26.24; 152.32 -> 24.83 | Helped |
| Use COLD-Attack instead of AutoDAN-Zhu for continuation-constrained suffix generation | LLM jailbreak prompts · AdvBench subset, continuation constraint, evaluated LLMs | ASR-G: 84.00 -> 86.00; 80.00 -> 84.00; 84.00 -> 90.00; 68.00 -> 66.00 | Mixed |
| Use COLD-Attack instead of PRISM for paraphrase-constrained prompts | LLM jailbreak prompts · AdvBench subset, paraphrasing constraint, Llama2 | ASR: 4.00 -> 86.00 | Helped |
| Use COLD-Attack instead of PRISM for paraphrase-constrained prompts | LLM jailbreak prompts · AdvBench subset, paraphrasing constraint, Llama2 | ASR-G: 2.00 -> 74.00 | Helped |
| Add negative sentiment steering rather than positive sentiment steering | COLD-Attack paraphrase prompts · AdvBench subset, sentiment steering, Guanaco | ASR-G: 56.00 -> 80.00 | Helped |
| Add negative sentiment steering rather than positive sentiment steering | COLD-Attack paraphrase prompts · AdvBench subset, sentiment steering, Mistral | ASR-G: 60.00 -> 90.00 | Helped |
| Add positive sentiment steering rather than negative sentiment steering | COLD-Attack paraphrase prompts · AdvBench subset, sentiment steering, Llama2 | ASR-G: 40.00 -> 58.00 | Helped |
| Insert COLD-Attack between the query and control prompt instead of using the prompt-only input | Llama2 · AdvBench subset, position constraint, lexical control | ASR-G: 24.00 -> 86.00 | Helped |
| Insert COLD-Attack between the query and control prompt instead of using the prompt-only input | Llama2 · AdvBench subset, position constraint, format control | ASR-G: 8.00 -> 86.00 | Helped |
| Use COLD-Attack instead of GCG for position-constrained prompts | Llama2 · AdvBench subset, position constraint, sentiment | PPL: 2587.90 -> 59.53 | Helped |
| Use COLD-Attack instead of AutoDAN-Zhu for position-constrained prompts | Llama2 · AdvBench subset, position constraint, sentiment | ASR-G: 72.00 -> 88.00 | Helped |
Surprising findings
- GCG and GCG-reg report 100% ASR on several models largely because early termination stops optimization once no refusal phrases appear, so substring ASR is inflated.
- Sentiment steering effects on attack success are model-dependent: negative sentiment helps against Mistral and Guanaco, while positive sentiment helps against Llama2.
Limitations
- Relies on white-box gradient access for the main attacks; transfer to GPT-3.5/GPT-4 is much lower (about 16-36% ASR-G).
- Substring ASR and GPT-4-based ASR-G can disagree, and the paper notes off-topic outputs for some baselines, so metrics are imperfect.
- Attack success only evaluated on a 50-instruction AdvBench subset in main experiments.
Experimental setup
Datasets: AdvBench
Benchmarks: AdvBench
Metrics: ASR ASR-G PPL Succ BLEU ROUGE BERTScore DNS ADN Self-BLEU
Baselines: UAT GBDA PEZ GCG GCG-reg AutoDAN-Zhu AutoDAN-Liu PRISM PAWS GPT-4 paraphrase
Builds on: COLD GCG AutoDAN Langevin dynamics Universal adversarial triggers
Closest papers by meaning
- GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs (NeurIPS 2025 · similarity 0.89)
- An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks (ICML 2025 · similarity 0.89)
- Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (ICLR 2024 · similarity 0.88)
- Weak-to-Strong Jailbreaking on Large Language Models (ICML 2025 · similarity 0.88)
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.88)
- Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimization (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.