Research map / Adversarial robustness and security / Attacks on text and graphs
An LLM can Fool Itself: A Prompt-Based Adversarial Attack
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PromptAttack, which prompts an LLM to generate its own adversarial text variants (via character, word, and sentence perturbation instructions, a fidelity filter, and few-shot and ensemble strategies) to audit black-box LLM robustness on GLUE.
Problem
Existing adversarial robustness benchmarks for LLMs (AdvGLUE, AdvGLUE++) are generated against BERT-based or open-source models and transferred to black-box LLMs like GPT-3.5, so they may not genuinely measure a victim LLM's robustness and are computationally costly to build.
Approach
Build an attack prompt (original input, attack objective, attack guidance with nine perturbation instructions) so the victim LLM itself rewrites the input to fool itself, requiring only API queries; filter outputs by word modification ratio and BERTScore, and ensemble across perturbation levels.
Key results
- PromptAttack-FS-EN raises average ASR on GPT-3.5 across GLUE tasks to 48.34% vs 25.51% for AdvGLUE and 6.64% for AdvGLUE++.
- On SST-2 with GPT-3.5, ASR rises from 33.04% (AdvGLUE) to 75.23% (PromptAttack-FS-EN).
- On Llama2-13B, PromptAttack-EN reaches 63.20% average ASR versus 40.76% for AdvGLUE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace AdvGLUE samples with PromptAttack-FS-EN samples | GPT-3.5 · SST-2 | ASR: 33.04 -> 75.23 | Helped |
| Replace AdvGLUE samples with PromptAttack-FS-EN samples | GPT-3.5 · QQP | ASR: 14.76 -> 39.61 | Helped |
| Add few-shot attack guidance to PromptAttack-EN (ablation) | GPT-3.5 · GLUE Avg | ASR: 42.54 -> 48.34 | Helped |
| Add few-shot attack guidance to PromptAttack-EN (ablation) | Llama2-7B · GLUE Avg | ASR: 50.82 -> 40.08 | Hurt |
| Add few-shot attack guidance to PromptAttack-EN (ablation) | Llama2-13B · GLUE Avg | ASR: 63.20 -> 61.80 | Hurt |
| Add few-shot attack guidance to character perturbation C1 (ablation) | GPT-3.5 PromptAttack · GLUE Avg | ASR: 10.09 -> 10.05 | No clear effect |
| Add few-shot attack guidance to character perturbation C2 (ablation) | GPT-3.5 PromptAttack · GLUE Avg | ASR: 14.60 -> 16.58 | Helped |
| Add few-shot attack guidance to character perturbation C3 (ablation) | GPT-3.5 PromptAttack · GLUE Avg | ASR: 8.85 -> 10.37 | Helped |
| Add few-shot attack guidance to word perturbation W1 (ablation) | GPT-3.5 PromptAttack · GLUE Avg | ASR: 3.23 -> 6.76 | Helped |
| Add few-shot attack guidance to word perturbation W2 (ablation) | GPT-3.5 PromptAttack · GLUE Avg | ASR: 5.31 -> 8.41 | Helped |
| Add few-shot attack guidance to word perturbation W3 (ablation) | GPT-3.5 PromptAttack · GLUE Avg | ASR: 9.67 -> 13.36 | Helped |
| Add few-shot attack guidance to sentence perturbation S1 (ablation) | GPT-3.5 PromptAttack · GLUE Avg | ASR: 13.70 -> 15.23 | Helped |
Surprising findings
- A simple emoji appended to the input (e.g., ':)') can make GPT-3.5 flip its prediction.
- Few-shot attack guidance lowers ASR against Llama2 but raises it against GPT-3.5, attributed to the smaller model's weaker comprehension.
Limitations
- Results depend on the fidelity filter thresholds (word modification ratio 15%, BERTScore threshold per task), and unfiltered ASR is argued to be meaningless.
- Evaluated only on GLUE classification tasks and a limited set of LLMs (Llama2 and GPT-3.5).
Experimental setup
Datasets: GLUE SST-2 QQP MNLI RTE QNLI AdvGLUE AdvGLUE++
Benchmarks: SST-2 QQP MNLI-m MNLI-mm RTE QNLI
Metrics: attack success rate BERTScore word modification ratio
Baselines: AdvGLUE AdvGLUE++
Builds on: AdvGLUE AdvGLUE++ PromptBench TextBugger TextFooler
Closest papers by meaning
- AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs (ICML 2025 · similarity 0.90)
- On the Robustness of Verbal Confidence of LLMs in Adversarial Attacks (NeurIPS 2025 · similarity 0.90)
- Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space (NeurIPS 2024 · similarity 0.89)
- Improved Generation of Adversarial Examples Against Safety-aligned LLMs (NeurIPS 2024 · similarity 0.89)
- Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts? (ICLR 2025 · similarity 0.89)
- Fight Back Against Jailbreaking via Prompt Adversarial Tuning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.