Research map / Adversarial robustness and security / Attacks on text and graphs
GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GASP, a fully black-box method that trains a suffix-generator LLM and uses latent Bayesian optimization with GASPEval feedback plus ORPO fine-tuning to produce readable adversarial suffixes that jailbreak LLMs.
Problem
Optimization-based jailbreak suffixes are often unnatural and costly to find via discrete token search, and many require white-box access; the paper targets efficient, readable, black-box suffix jailbreaks.
Approach
Trains a SuffixLLM (AdvPrompter-style) on AdvSuffixes, then searches its final-layer latent space with latent Bayesian optimization (GP surrogate and acquisition function) using a 21-criterion LLM judge (GASPEval) on target responses, and refines the generator with ORPO on the ranked suffixes.
Key results
- GASP achieves ASR@10 of 68% on LLaMA-3.1-8B and 64% on LLaMA-2-7B under GASPEval on the out-of-distribution set, versus lower values for baselines such as AdvPrompter (17% and 7%).
- GASP reduces training time by 1.75x compared to AdvPrompter.
- GASP reaches 47% ASR against GPT-4o and 59% against Claude-3.7-Sonnet at a reported total cost of $5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GASP instead of AdvPrompter | black-box jailbreak attack · OOD harmful prompts across five target models | ASR@10 / ASR@1 (GASPEval) | Mixed |
| Add latent Bayesian optimization (LBO) to suffix generation (ablation) | SuffixLLM · Mistral-7B-v0.3 and Falcon-7B | ASR | Helped |
| Guide LBO with GASPEval rather than StrongREJECT (ablation) | GASP · LBO-guided jailbreak generation | ASR | Helped |
| Use ORPO rather than SFT for generator fine-tuning (ablation) | GASP · Suffix generation | ASR | Helped |
| Use ORPO rather than SFT for generator fine-tuning (ablation) | GASP · Suffix generation | suffix coherence | Helped |
| Use GASP instead of AdvPrompter | black-box jailbreak attack · Training | training time: 1.75× | Helped |
| Use GASP instead of GCG and AutoDAN | jailbreak prompt generation · Readability evaluation | readability | Helped |
| Use GASP instead of baseline suffix generation (ablation) | SuffixLLM · LBO-guided suffix generation | ASR | Helped |
Surprising findings
- Keyword matching overstates success for several heuristic/optimization baselines (e.g., AutoDAN 100% ASR@10 by keywords vs. 62% by GASPEval on Mistral), indicating evaluator choice strongly changes rankings.
- Using the stricter StrongREJECT as the LBO guiding objective notably reduces ASR compared with GASPEval.
Limitations
- Results depend on a custom LLM-based evaluator (GASPEval) whose reliability is not independently validated beyond comparisons.
- Reported results use median of 3 runs without error bars, limiting statistical significance assessment.
- Strong performance drops on some models (e.g., LLaMA-2-7B keyword ASR 0% at ASR@1), and pretraining relies on uncensored-model-generated suffixes.
Experimental setup
Datasets: AdvSuffixes AdvBench HarmBench
Benchmarks: AdvBench HarmBench AdvSuffixes out-of-distribution test (100 prompts)
Metrics: ASR@1 ASR@10 readability score training time inference time
Baselines: GCG AutoDAN AdvPrompter PAIR TAP ICA
Builds on: AdvPrompter ORPO GCG AutoDAN
Closest papers by meaning
- LARGO: Latent Adversarial Reflection through Gradient Optimization for Jailbreaking LLMs (NeurIPS 2025 · similarity 0.91)
- VERA: Variational Inference Framework for Jailbreaking Large Language Models (NeurIPS 2025 · similarity 0.90)
- Fight Back Against Jailbreaking via Prompt Adversarial Tuning (NeurIPS 2024 · similarity 0.90)
- AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs (ICML 2025 · similarity 0.90)
- JailPO: A Novel Black-Box Jailbreak Framework via Preference Optimization Against Aligned LLMs (AAAI 2025 · similarity 0.90)
- Best-of-N Jailbreaking (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.