Research map / Adversarial robustness and security / Attacks on text and graphs
Adversarial prompts for language models
Attack and defend language models with adversarial prompts and robustness benchmarks.
98 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2019–2026) · an approach in Attacks on text and graphs, keyword cluster “llms · prompt · jailbreak”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Leveling off Several papers from 2025–26 still report on 1 tracked benchmark, but the best result has barely moved since 2024.
17 papers in 2023–24 and 65 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 6.1% to 13.4%.
What looks settled
- MT-Bench (score): best up to 2024 8, best in 2025–26 7.38 from 3 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2019 · Improving Neural Language Modeling via Adversarial Training (ICML 2019 · 53 citations)
- 2020 · Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment (AAAI 2020 · 887 citations)
- 2021 · Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models (NeurIPS 2021 · 72 citations)
- 2022 · The King Is Naked: On the Notion of Robustness for Natural Language Processing (AAAI 2022 · 15 citations)
- 2023 · Are aligned neural networks adversarially aligned? (NeurIPS 2023 · 25 citations)
- 2024 · Training Socially Aligned Language Models on Simulated Social Interactions (ICLR 2024 · 16 citations)
- 2025 · AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs (ICML 2025 · 4 citations)
- 2026 · Disentangling Adversarial Prompts: A Semantic-Graph Defense for Robust LLM Security (AAAI 2026 · 1 citation)
Earliest papers here
- Improving Neural Language Modeling via Adversarial Training (ICML 2019 · 53 citations)
- Re-Evaluating ADEM: A Deeper Look at Scoring Dialogue Responses (AAAI 2019)
- Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment (AAAI 2020 · 887 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
MT-Bench · score (higher is better)
4 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 8 | Improving Alignment and Robustness with Circuit Breakers (NeurIPS 2024 · 11 citations) |
What has been tried
57 papers here report 659 measured changes (a component added, removed or swapped, compared with a baseline): helped 61%, no clear effect 11%, hurt 24%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add Realignment (RA) to IL + SC | HH-A · Alignment | 6.59 -> 8.23 | Helped | Training Socially Aligned Language Models on Simulated Social Interactions (ICLR 2024 · 16 citations) |
| Add few-shot attack guidance to PromptAttack-EN | GLUE Avg · ASR | 42.54 -> 48.34 | Helped | An LLM can Fool Itself: A Prompt-Based Adversarial Attack (ICLR 2024 · 13 citations) |
| Add circuit-breaker examples that bypass refusal mechanisms | 6 attacks (average) · attack success rate | 5.8 -> 2.5 | Helped | Improving Alignment and Robustness with Circuit Breakers (NeurIPS 2024 · 11 citations) |
| Remove adversarial attacks during CAPO training | GCG · ASR difference | -2.5 | No clear effect | Efficient Adversarial Training in LLMs with Continuous Attacks (NeurIPS 2024 · 10 citations) |
| remove parent-child similarity filter | Llama 2-chat 7B archive · ASR (GPT-4) | 0.92 ± 0.01 -> 0.99 ± 0.01 | Helped | Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts (NeurIPS 2024 · 7 citations) |
| Set top-p to 0.992 instead of 0.99 | All 8 deployed detectors · mean T@1%F | 0.0690 -> 0.0565 | Helped | Adversarial Paraphrasing: A Universal Attack for Humanizing AI-Generated Text (NeurIPS 2025 · 2 citations) |
| Remove refusal feature ablation during training (RT, p_RFA=0) | HarmBench, robustness across attacks · ASR | Hurt | Robust LLM safeguarding via refusal feature adversarial training (ICLR 2025 · 2 citations) | |
| add training languages that use the same script | Urdu · BLEU | +100.20% | Helped | Against All Odds: Overcoming Typology, Script, and Language Confusion in Multilingual Embedding Inversion Attacks (AAAI 2025 · 1 citation) |
| Remove erase loss | WMDP-bio · accuracy | 29.7 -> 64.8 | Hurt | Erasing Conceptual Knowledge from Language Models (NeurIPS 2025 · 1 citation) |
| Remove negative text from the CLIP attack objective | GPT-4V targeted caption generation · Targeted cap. | 71% to 46% | Hurt | Dissecting Adversarial Robustness of Multimodal LM Agents (ICLR 2025 · 1 citation) |
| Reduce candidate positions from n=20 to n=1 (Charmer-Fast) | TextAttack benchmarks · ASR (%) | BERT AG-News 98.51 -> 95.86; BERT RTE 97.01 -> 89.55; BERT SST-2 100.00 -> 100.00 | Mixed | Revisiting Character-level Adversarial Attacks for Language Models (ICML 2024 · 1 citation) |
| Add a random generation to the affirmative seed | HarmBench standard behaviors · ASR@512 | 0.35 -> 0.36 | No clear effect | REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective (ICML 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 35 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- AdvBench · 8 of 35 papers
- HarmBench · 6 of 35 papers
- MMLU · 5 of 35 papers
- Alpaca · 4 of 35 papers
- MT-Bench · 3 of 35 papers
- XSTest · 3 of 35 papers
- CIFAR-10 · 2 of 35 papers
- ImageNet · 2 of 35 papers
Baselines
- GCG · 5 of 35 papers
- PAIR · 4 of 35 papers
- AutoDAN · 2 of 35 papers
- BEAST · 2 of 35 papers
- R2D2 · 2 of 35 papers
- CAT · 2 of 35 papers
- LAT · 2 of 35 papers
- SmoothLLM · 2 of 35 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 82 papers; from OpenAlex and the papers' affiliations).
- Google · 6 papers
- National University of Singapore · 4 papers
- ETHZ - ETH Zurich · 4 papers
- Google DeepMind · 3 papers
- Meta · 3 papers
- Carnegie Mellon University · 3 papers
- Nanyang Technological University · 3 papers
- University of Technology Sydney · 2 papers
- ShanghaiTech University · 2 papers
- North Carolina State University · 2 papers
Use it
Download BibTeX of 34 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- LLM jailbreak and injection defense (113 papers)
- Vision-language model robustness (96 papers)
Elsewhere in the topic
- Membership inference and inversion (118 papers)
- Defenses against universal perturbations (97 papers)
- Attacks and defenses in sequential settings (96 papers)
- Diffusion-based adversarial purification (78 papers)
- Biologically inspired robust vision (77 papers)
- Graph neural network robustness (71 papers)
- Adversarial perturbations beyond images (69 papers)
- Physical adversarial patches and camouflage (56 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).