Research map / Large language models / Alignment and preferences
Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Gradient Cuff, a training-free two-step jailbreak detector that thresholds a sampling-based refusal loss and its zeroth-order estimated gradient norm on the protected LLM.
Problem
Jailbreak attacks bypass LLM safety alignment, and existing unsupervised defenses either miss many attack types or heavily reject benign queries.
Approach
Defines a refusal loss (probability of non-refusal estimated by sampling responses and keyword matching) and observes that malicious queries have lower loss values and steeper landscapes; detection thresholds the loss and then the zeroth-order gradient norm of the loss in embedding space.
Key results
- Gradient Cuff reaches average TPR 0.883 on LLaMA2-7B-Chat (FPR 0.022) and 0.743 on Vicuna-7B-V1.5 (FPR 0.034) across six jailbreaks at 5% benign-refusal threshold.
- Average attack success rate on Vicuna-7B-V1.5 is reduced from 76.7% to 25.7% per the paper's Figure 1(d) claim.
- Self-Reminder combined with Gradient Cuff raises malicious refusal by 34.96% on Vicuna-7B-V1.5 (0.472 to 0.637 is Self-Reminder to Self-Reminder (GC)).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gradient Cuff instead of SmoothLLM | LLaMA2-7B-Chat · Malicious jailbreak queries, average across 6 attacks | TPR: 0.763±0.017 -> 0.883±0.013 | Helped |
| Use Gradient Cuff instead of SmoothLLM | Vicuna-7B-V1.5 · Malicious jailbreak queries, average across 6 attacks | TPR: 0.413±0.020 -> 0.743±0.018 | Helped |
| Add Gradient Cuff to Self-Reminder (ablation) | LLaMA2-7B-Chat · Malicious jailbreak queries, average across 6 attacks | TPR: 0.820±0.016 -> 0.920±0.011 | Helped |
| Add Gradient Cuff to Self-Reminder (ablation) | Vicuna-7B-V1.5 · Malicious jailbreak queries, average across 6 attacks | TPR: 0.472±0.020 -> 0.637±0.020 | Helped |
| Add the gradient-norm rejection stage (ablation) | Gradient Cuff on LLaMA2-7B-Chat · Malicious jailbreak queries, average across 6 attacks | TPR: 0.698±0.019 -> 0.797±0.016 | Helped |
| Add the gradient-norm rejection stage (ablation) | Gradient Cuff on Vicuna-7B-V1.5 · Malicious jailbreak queries, average across 6 attacks | TPR: 0.296±0.019 -> 0.536±0.020 | Helped |
| Add the gradient-norm rejection stage (ablation) | Gradient Cuff on LLaMA2-7B-Chat · Benign user queries | FPR: 0.012±0.011 -> 0.012±0.011 | No clear effect |
| Add the gradient-norm rejection stage (ablation) | Gradient Cuff on Vicuna-7B-V1.5 · Benign user queries | FPR: 0.008±0.009 -> 0.010±0.010 | No clear effect |
| Increase the second-stage threshold setting from σ=1% to σ=5% (ablation) | Gradient Cuff on LLaMA2-7B-Chat · Malicious jailbreak queries, average across 6 attacks | TPR: 0.797±0.016 -> 0.883±0.013 | Helped |
| Increase the second-stage threshold setting from σ=1% to σ=5% (ablation) | Gradient Cuff on Vicuna-7B-V1.5 · Malicious jailbreak queries, average across 6 attacks | TPR: 0.536±0.020 -> 0.743±0.018 | Helped |
| Increase query budget using fixed-N rather than fixed-P (ablation) | Gradient Cuff · Average refusal rate across 6 jailbreak datasets | TPR improvement: 0.137 v.s. 0.026 | Helped |
Surprising findings
- Erase-Check achieves high malicious refusal but rejects about 86-99% of benign queries, making it impractical.
- Adaptive PAIR cuts Gradient Cuff's refusal rate on Vicuna-7B-V1.5 from 0.694 to 0.356, showing adaptive attacks can weaken the defense.
Limitations
- Requires white-box access to model weights and embeddings, so applicability to closed-source LLMs is limited to developer-side deployment.
- Adaptive attacks (especially PAIR and TAP) reduce detection performance.
- High inference cost from hundreds of LLM queries per input (q = N(P+1)).
Experimental setup
Datasets: AdvBench LMSYS Chatbot Arena conversations
Benchmarks: MMLU GCG AutoDAN PAIR TAP Base64 LRL
Metrics: TPR (refusal rate on malicious queries) FPR (refusal rate on benign queries) attack success rate MMLU accuracy
Baselines: PPL SmoothLLM Erase-Check Self-Reminder LLaMA-Guard Safe-Decoding
Builds on: SmoothLLM GCG Self-Reminder PAIR Erase-Check
Closest papers by meaning
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.91)
- JailPO: A Novel Black-Box Jailbreak Framework via Preference Optimization Against Aligned LLMs (AAAI 2025 · similarity 0.90)
- Adversarial Reasoning at Jailbreaking Time (ICML 2025 · similarity 0.90)
- Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (ICLR 2024 · similarity 0.89)
- Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models (AAAI 2025 · similarity 0.89)
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.