Research map / Large language models / AI and society
Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reports the 2024 SaTML LLM capture-the-flag prompt-injection competition, where teams built defenses protecting a secret in the system prompt and others attacked them, and releases a 137k-chat multi-turn attack dataset.
Problem
System prompt confidentiality and instruction-following robustness of LLM applications against prompt injection and secret extraction attacks, which lack established defenses.
Approach
A two-phase capture-the-flag competition in which teams first design defenses (system prompt, Python filter, LLM filter) for a secret and then attack other teams' defenses, with scoring that rewards fewer chats and early breaks; the resulting chat logs are released as a dataset.
Key results
- The competition had 163 registered teams; 72 defenses were submitted by 39 teams, and 44 were accepted after utility evaluation.
- The attack phase produced 137,063 unique chats, with 35 teams extracting at least one secret.
- All defenses were bypassed at least once; only 5,461 chats (4%) were successful extractions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-turn rather than single-turn attacks | Competition attacks · Secret extraction | successful attack rate: 67.8% vs 82.5% (chats with 1 attacker message); 12.6% vs 4.1% (chats with 4-7 attacker messages) | Helped |
| adapt attacks to assumptions about the specific defense | Competition attacks · Submitted defenses | secret extraction | Helped |
| use decoy secrets in the defense prompt | Prompt-injection defenses · Secret extraction | defense robustness | Mixed |
| use threats about consequences for revealing the secret (ablation) | RSLLM defense · Secret extraction | defense effectiveness | No clear effect |
| insert random text after the secret (ablation) | RSLLM defense · Secret extraction | defense effectiveness | No clear effect |
| use random capitalization with LLM restoration (ablation) | RSLLM defense on Llama-2 70B · Benign utility | utility | Hurt |
| replace random capitalization with lowercase output (ablation) | RSLLM defense on Llama-2 70B · Benign utility | utility | Helped |
| use GPT-3.5 rather than Llama-2 70B | Competition defenses · Secret extraction | defense robustness: 6 out of the 7 | Mixed |
Surprising findings
- More capable GPT-3.5 was not more robust as a defense base; 6 of the top 7 defenses used Llama-2.
- Successful attacks relied more on multi-turn chats: 33% of successful chats used more than one user message versus 18% of unsuccessful ones.
Limitations
- Dataset scope is narrow, focusing only on secret extraction with mostly six-character secrets.
- Some success labels may be unreliable due to secrets split across chats or unsubmitted extractions.
- Only black-box prompting and filtering defenses were evaluated, not white-box methods.
Experimental setup
Datasets: MMLU ARB
Benchmarks: MMLU ARB
Metrics: secret extraction success rate chats per extracted secret utility score
Baselines: TensorTrust Gandalf
Builds on: TensorTrust Gandalf
Closest papers by meaning
- NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security (NeurIPS 2024 · similarity 0.85)
- EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities (ICML 2025 · similarity 0.84)
- CoreGuard: Safeguarding Foundational Capabilities of LLMs Against Model Stealing in Edge Deployment (NeurIPS 2025 · similarity 0.84)
- On Evaluating the Durability of Safeguards for Open-Weight LLMs (ICLR 2025 · similarity 0.84)
- On Calibration of LLM-based Guard Models for Reliable Content Moderation (ICLR 2025 · similarity 0.84)
- Bits Leaked per Query: Information-Theoretic Bounds for Adversarial Attacks on LLMs (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.