Research map / Large language models / Language, safety and interpretability
Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Releases a large dataset of human-written prompt injection attacks and defenses collected from the Tensor Trust web game, builds two robustness benchmarks (hijacking and extraction) and evaluates eight LLMs on them.
Problem
LLM applications are vulnerable to prompt injection, but existing adversarial datasets focus on jailbreaks or are automatically generated and hard to interpret; there is little human-generated prompt injection data with defenses.
Approach
Crowdsource attacks and defenses through an online game where players defend a secret access code with prompts and others attack; the resulting data is used to build filtered robustness benchmarks for prompt hijacking and prompt extraction, with adversarial filtering against reference models.
Key results
- GPT-4 achieves 84.3% hijacking robustness rate with 81.7% defense validity, beating other evaluated models on the hijacking benchmark.
- On the extraction benchmark GPT-4 reaches 69.1% extraction robustness rate, while Llama-2-7b-chat has 44.9% ERR and 17.4% DV.
- Zero-shot GPT-4 detects prompt extraction with 97.0% precision and 84.3% recall, while GPT-3.5 Turbo reaches only 13.0% recall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4 instead of GPT-3.5 Turbo | Tensor Trust prompt hijacking benchmark · Prompt hijacking robustness benchmark | HRR: 18.4% -> 84.3% | Helped |
| Use GPT-4 instead of GPT-3.5 Turbo | Tensor Trust prompt hijacking benchmark · Prompt hijacking robustness benchmark | DV: 89.2% -> 81.7% | Hurt |
| Use GPT-4 instead of GPT-3.5 Turbo | Tensor Trust prompt extraction benchmark · Prompt extraction robustness benchmark | ERR: 12.3% -> 69.1% | Helped |
| Use GPT-4 instead of GPT-3.5 Turbo | Tensor Trust prompt extraction benchmark · Prompt extraction robustness benchmark | DV: 91.1% -> 89.5% | Hurt |
| Use GPT-4 instead of GPT-3.5 Turbo | Zero-shot prompt extraction detection · Prompt extraction detection dataset | Accuracy: 56.5% -> 90.9% | Helped |
| Use GPT-4 instead of GPT-3.5 Turbo | Zero-shot prompt extraction detection · Prompt extraction detection dataset | Recall: 13.0% -> 84.3% | Helped |
| Use GPT-4 instead of GPT-3.5 Turbo | Zero-shot prompt extraction detection · Prompt extraction detection dataset | Precision: 100.0% -> 97.0% | Mixed |
| Change the message-role assignment (ablation) | GPT-3.5 Turbo in Tensor Trust · Prompt hijacking and extraction benchmarks | Performance across metrics | No clear effect |
| Mark the attack/access-code message as system instead of user (user/system/user) (ablation) | GPT-3.5 Turbo in Tensor Trust · Prompt hijacking and extraction benchmarks | Performance across metrics | Hurt |
Surprising findings
- Claude-instant-1.2 and LLaMA models show trade-offs where spurious refusals of defenses raise hijacking robustness but lower defense validity.
- Repeated-character and rare-token attacks (e.g. 'artisanlib') transfer to deployed applications and can elicit jailbreak-like behavior.
Limitations
- Attacks are constrained to a game setting with an access-code objective and ToS-based data filtering, so they may not represent all real-world attacks.
- Benchmarks depend on adversarial filtering against three reference models, which biases results.
- Prompt extraction detection by GPT-3.5 Turbo is near random, and robust detection remains open.
Experimental setup
Datasets: Tensor Trust game attacks and defenses
Benchmarks: Tensor Trust prompt hijacking benchmark Tensor Trust prompt extraction benchmark Tensor Trust prompt extraction detection dataset
Metrics: Hijacking Robustness Rate Extraction Robustness Rate Defense Validity precision recall accuracy
Baselines: GPT-3.5 Turbo GPT-4 Claude-instant-1.2 Claude-2.0 PaLM 2 Llama-2-7b-chat Llama-2-13b-chat Llama-2-70b-chat CodeLlama-34b-Instruct
Builds on: LDA topic modeling minhash locality-sensitive hashing
Closest papers by meaning
- Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space (NeurIPS 2024 · similarity 0.88)
- Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks (NeurIPS 2026 · similarity 0.88)
- An LLM can Fool Itself: A Prompt-Based Adversarial Attack (ICLR 2024 · similarity 0.88)
- ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents (ICLR 2026 · similarity 0.88)
- Persistent Pre-training Poisoning of LLMs (ICLR 2025 · similarity 0.87)
- Many-shot Jailbreaking (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.