Research map / Adversarial robustness and security / Black-box and transfer attacks
Query-Based Adversarial Prompt Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Greedy Coordinate Query (GCQ), a query-based adversarial suffix attack that uses API access to a remote LLM (and a local proxy model, or none) to elicit exact harmful strings and evade content moderation classifiers.
Problem
Transfer-based adversarial attacks on aligned language models are weak at targeted behavior (exact harmful strings) and depend on a high-quality surrogate model; white-box attacks need full model access.
Approach
Split each GCG iteration into a gradient-based (proxy-model) candidate filter followed by selection using true loss queries to the target model via API; a best-first buffer search replaces brute-force GCG, and a variant drops the proxy entirely, making it a pure query-only attack.
Key results
- On gpt-3.5-turbo-instruct, GCQ reaches 79.6% attack success rate on 574 harmful strings with at most $0.10 per target, rising to 86.0% at $0.20 per target; total cost $80.
- Universal suffix (20 tokens) reaches 99.2% attack success rate evading OpenAI text-moderation-007 on validation strings; nonuniversal 20-token suffixes reach 91.4% after GCQ iterations.
- In the Vicuna 1.3 7B harmful-string setting, GCQ with a 13B proxy reaches 0.388 success vs 0.000 for GCG pure transfer; AutoDAN achieves 0.002 at best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace pure transfer optimization with target-model queries using GCQ | GCG Pure Transfer on harmful strings · Vicuna 1.3 7B | success rate: 0.000 -> 0.791 | Helped |
| Replace pure transfer optimization with target-model queries using GCQ | GCG Pure Transfer on harmful strings · Vicuna 1.3 7B target, Vicuna 1.3 13B proxy | success rate: 0.000 -> 0.388 | Helped |
| Increase the per-target budget from $0.10 to $0.20 | GCQ on GPT-3.5 Turbo Instruct · Harmful strings | attack success rate: 79.6\% -> 86.0\% | Helped |
| Use a 40-token prompt and raise the per-target budget to $2 | GCQ on GPT-3.5 Turbo Instruct · Previously failed harmful strings longer than 20 tokens | attack success rate: 100\% | Helped |
| Add target-repeat initialization instead of using uniform random prompts | GCQ on GPT-3.5 Turbo Instruct · Harmful strings | immediate target-string success: 28\% | Helped |
| Use random initialization instead of target-repeat initialization (ablation) | GCQ on GPT-3.5 Turbo Instruct · First 20 harmful strings | successful targets: only twice | Hurt |
| Use the optimized single-position search rather than original GCG candidate updates (ablation) | GCG on Vicuna 7B · Proxy-free harmful strings | query efficiency: 2× more query-efficient | Helped |
| Remove gradient information and sample token replacements at random (ablation) | Optimized GCG on Vicuna 7B · Proxy-free harmful strings | attack success rate: about 30\% | Helped |
| Short-circuit cumulative logprob computation when a candidate can be discarded (ablation) | GCQ · Harmful-string attack cost | cost: approximately 30\% | Helped |
| Re-tokenize prompts before sending them to the API (ablation) | GCQ · Harmful-string generation | success rate | No clear effect |
Surprising findings
- Initializing the prompt by repeating the target string solved 28% of harmful strings immediately, due to the model continuing repetitions.
- Random initialization made the attack succeed on only 2 of the first 20 strings, showing initialization is crucial in the low-cost regime.
Limitations
- Attack relies on logprob access (reconstructed via logit_bias/top-5 logprobs), which OpenAI later removed, making attacks much costlier.
- Success drops sharply when target string length approaches or exceeds prompt length.
- Nondeterministic API outputs: only about 90% of solved prompts reproduce on re-query.
Experimental setup
Datasets: harmful strings dataset (574 strings)
Benchmarks: harmful strings dataset OpenAI text-moderation-007 evaluation on harmful strings Llama Guard 7B evaluation on harmful strings
Metrics: attack success rate cost in USD number of queries/requests
Baselines: GCG GCG Pure Transfer AutoDAN GA AutoDAN HGA
Builds on: GCG AutoPrompt AutoDAN
Closest papers by meaning
- Improved Generation of Adversarial Examples Against Safety-aligned LLMs (NeurIPS 2024 · similarity 0.90)
- Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.87)
- Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts? (ICLR 2025 · similarity 0.87)
- An LLM can Fool Itself: A Prompt-Based Adversarial Attack (ICLR 2024 · similarity 0.87)
- Fight Back Against Jailbreaking via Prompt Adversarial Tuning (NeurIPS 2024 · similarity 0.86)
- Revisiting Character-level Adversarial Attacks for Language Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.