Research map / Large language models / Language, safety and interpretability
Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Token Highlighter, a training-free jailbreak defense that uses the gradient norm of an Affirmation Loss to find jailbreak-critical query tokens and shrinks their embeddings (Soft Removal) before generation.
Problem
Aligned LLMs remain vulnerable to jailbreak prompts (token-level and prompt-level), while existing defenses either fail on fluent prompts, have high false positive rates, or require multiple LLM queries at high cost.
Approach
Define an Affirmation Loss (negative log-likelihood of a pre-set affirmative reply like "Sure, I'd like to help you with this.") and use per-token gradient norms of it to select the top-alpha fraction of query tokens; then multiply their embeddings by beta in (0,1) ("Soft Removal") instead of deleting them.
Key results
- On Vicuna-7B-V1.5, Token Highlighter reduces average ASR from 0.730 to 0.142 while keeping AlpacaEval win rate at 0.698 (vs 0.772 undefended).
- The best baseline Gradient Cuff reduces Vicuna ASR only to 0.243 (per text; its table reports average 0.408).
- Combining with Self Reminder as Self Reminder (TH) at beta=0.3 gives ASR 0.023 with win rate 0.536.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add gradient-based Critical Set selection rather than selecting soft-removal tokens randomly (ablation) | Token Highlighter · Vicuna-7B-V1.5, jailbreak attacks | average ASR: 0.550 -> 0.142 | Helped |
| add gradient-based Critical Set selection rather than selecting soft-removal tokens randomly (ablation) | Token Highlighter · Vicuna-7B-V1.5, AlpacaEval | Win Rate: 0.743 -> 0.698 | No clear effect |
| add gradient-based Critical Set selection rather than selecting soft-removal tokens randomly (ablation) | Token Highlighter · LLaMA-2-7B-Chat, jailbreak attacks | average ASR: 0.030 -> 0.018 | Mixed |
| increase β from 0.3 to 0.5 at α=0.25 (ablation) | Token Highlighter on Vicuna-7B-V1.5 · jailbreak attacks | average ASR: 0.142 -> 0.597 | Hurt |
| increase β from 0.3 to 0.5 at α=0.25 (ablation) | Token Highlighter on Vicuna-7B-V1.5 · AlpacaEval | Win Rate: 0.698 -> 0.772 | Helped |
| increase α from 0.25 to 0.5 at β=0.3 (ablation) | Token Highlighter on Vicuna-7B-V1.5 · jailbreak attacks | average ASR: 0.142 -> 0.080 | Helped |
| increase α from 0.25 to 0.5 at β=0.3 (ablation) | Token Highlighter on Vicuna-7B-V1.5 · AlpacaEval | Win Rate: 0.698 -> 0.588 | Hurt |
| use a different affirmative sentence, “Sure, here is the what I've got for you.” (ablation) | Token Highlighter on Vicuna-7B-V1.5 · jailbreak attacks | ASR: 0.142 -> 0.143 | No clear effect |
| use a different affirmative sentence, “Here is” (ablation) | Token Highlighter on Vicuna-7B-V1.5 · jailbreak attacks | ASR: 0.142 -> 0.145 | No clear effect |
| combine Token Highlighter with Self Reminder using β=0.5 (ablation) | Self Reminder on Vicuna-7B-V1.5 · jailbreak attacks | ASR: 0.427 -> 0.362 | Helped |
| combine Token Highlighter with Self Reminder using β=0.3 (ablation) | Self Reminder on Vicuna-7B-V1.5 · jailbreak attacks | ASR: 0.427 -> 0.023 | Helped |
| combine Token Highlighter with Self Reminder using β=0.3 (ablation) | Self Reminder on Vicuna-7B-V1.5 · AlpacaEval | Win Rate: 0.684 -> 0.536 | Hurt |
Surprising findings
- Random Soft Removal keeps utility almost unchanged, suggesting beta and alpha matter more than which tokens are removed for utility.
Limitations
- Utility drops as alpha and beta are made more aggressive; trade-off between ASR and win rate.
- Adaptive attacks (Adaptive-GCG, Adaptive-TAP) raise ASR somewhat (up to 0.1 on Vicuna).
Experimental setup
Datasets: AdvBench AlpacaEval
Benchmarks: AdvBench AlpacaEval
Metrics: Attack Success Rate (ASR) Win Rate
Baselines: PPL Erase Check Gradient Cuff SmoothLLM Semantic Smoothing Self Reminder Random Soft Removal
Builds on: GCG Self Reminder SmoothLLM Gradient Cuff Erase Check
Closest papers by meaning
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.90)
- SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism (NeurIPS 2025 · similarity 0.89)
- Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs (NeurIPS 2024 · similarity 0.89)
- Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes (NeurIPS 2024 · similarity 0.89)
- JailPO: A Novel Black-Box Jailbreak Framework via Preference Optimization Against Aligned LLMs (AAAI 2025 · similarity 0.89)
- Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.