Research map / Adversarial robustness and security / Attacks on text and graphs
Gandalf the Red: Adaptive Security for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D-SEC, a dynamic security-utility threat model for LLM applications, and Gandalf, a crowd-sourced gamified red-teaming platform that yields a released 279k-prompt attack dataset used to study defense strategies.
Problem
Evaluations of defenses against prompt attacks on LLM applications ignore adaptive attackers and the usability cost that restrictive defenses impose on legitimate users, and benchmark attacks are often non-adaptive and easily blocked.
Approach
Model attackers and benign users separately in multi-transaction sessions, define an optimizable developer utility trading off security (Attacker Failure Rate) and utility (Session Completion Rate), and collect adaptive attacks via a gamified password-extraction game whose success label is objective (password extracted).
Key results
- Gandalf-RCT contains 279,675 prompt attacks across 36,286 sessions from 15,448 users.
- Defense-in-depth ('or' of C1, C2, C3 in level D) reduced the share of players beating the level to 2.9% (general), 6.3% (summarization), and 6.4% (topic).
- Adaptive blocking with threshold 3 on the combined defense blocks 75% of attacks missed by non-adaptive defenses.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Restrict the application to a summarization or topic domain rather than a general chatbot | Gandalf application · General, Summarization, and Topic | average levels solved per session: General: 1.95, 1.50, 1.73; Summarization: 1.27, 1.01, 1.28; Topic: 1.81, 1.12, 1.37 | Helped |
| Combine C1, C2, and C3 using or aggregation (ablation) | Gandalf defenses · General, Summarization, and Topic | players who beat level D: 2.9%, 6.3%, and 6.4% | Helped |
| Combine substring check, LLM checker, and strong prompt instead of using each defense individually (ablation) | Gandalf defenses · Gandalf-RCT attacks | attack blocking: 14.5% | Helped |
| Block the session after 3 flagged attempts (ablation) | Combined defense · Undetected attacks against the non-adaptive defense | attacks blocked: 75% | Helped |
| Add a system-prompt defense | LLM application · BasicUser and BorderlineUser | response character length | Hurt |
| Use a system-prompt defense | LLM application · Benign user requests | response content similarity | Hurt |
Surprising findings
- System prompt defenses reduce response length and alter content even when requests are not blocked, degrading utility invisibly to block-based metrics.
- Utility estimates differ substantially depending on which benign dataset (BasicUser vs BorderlineUser) is used.
Limitations
- Narrow password-extraction domain limits generalizability to other applications.
- Defenses were not equally optimized, so cross-type security comparisons are unreliable.
- Adaptive defenses studied are only simple time-based blocking thresholds.
Experimental setup
Datasets: Gandalf-RCT BasicUser BorderlineUser UltraChat-200k
Benchmarks: Gandalf-RCT BasicUser BorderlineUser
Metrics: Attacker Failure Rate Session Completion Rate Attacks per Exploit response length cosine similarity
Baselines: substring check LLM checker strong system prompt weak system prompt
Builds on: UltraChat
Closest papers by meaning
- Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks (NeurIPS 2024 · similarity 0.88)
- AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs (ICML 2025 · similarity 0.87)
- An LLM can Fool Itself: A Prompt-Based Adversarial Attack (ICLR 2024 · similarity 0.87)
- On Evaluating the Durability of Safeguards for Open-Weight LLMs (ICLR 2025 · similarity 0.86)
- WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models (NeurIPS 2024 · similarity 0.86)
- SafeHarbor: Defining Precise Decision Boundaries via Hierarchical Memory-Augmented Guardrail for LLM Agent Safety (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.