Research map / Large language models / Language, safety and interpretability
Prompt Injection as Role Confusion
Charles Ye, Jasmine Cui, Dylan Hadfield-Menell
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| strip model-characteristic reasoning style from forged CoT (destyling) (ablation) | CoT Forgery · StrongREJECT | attack success rate: 61% -> 10% | Hurt |
| make the forged justification absurd instead of plausible (ablation) | CoT Forgery · StrongREJECT | attack success rate: 60% vs. 63% | No clear effect |
| add forged reasoning to a standard tool-channel injection | agent exfiltration injection · agent data exfiltration (100 attacks) | attack success rate: 0-2% -> 56-70% | Improved |
| add forged CoT to the harmful query | GPT-5 family, o4-mini, gpt-oss · StrongREJECT | attack success rate: 0-4% -> 17-52% (GPT-5), >80% (gpt-oss, o4-mini) | Improved |
| strip all role tags from the conversation (ablation) | role probes · gardening conversation | CoTness of CoT-style text: 85% -> 83% | No clear effect |
| wrap whole conversation in <user> tags (ablation) | role probes · gardening conversation | CoTness of CoT-style text: 85% average CoTness, 2% Userness | No clear effect |
| re-wrap user-style text in <tool> tags (ablation) | four open models · OASST1 and ToxicChat conversations | Userness: Userness stays 76-88%, Toolness under 20% | No clear effect |
| increase CoTness of injected forgery (higher quantile) | CoT Forgery · StrongREJECT (626 attempts) | attack success rate: 9% -> 90% | Improved |
| increase Userness of injected command (higher quantile) | agent prompt injection · agent hijacking (1000 attempts) | attack success rate: 2% -> 70% | Improved |
| declare the role in plain text in the injection template (ablation) | agent prompt injection · 212 injection templates | Userness: 24% of templates make up 100% of top 5% | Improved |
| destyle the forged CoT (ablation) | CoT Forgery · StrongREJECT (626 attempts) | CoTness of forged text: 79% -> 29% | Hurt |
Closest papers by meaning
- The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them) (ICML 2025 · similarity 0.89)
- ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agents (NeurIPS 2026 · similarity 0.86)
- Concept Incongruence: An Exploration of Time and Death in Role Playing (NeurIPS 2025 · similarity 0.85)
- ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents (ICLR 2026 · similarity 0.85)
- Revisiting "Edit Away and My Face Will not Stay: Personal Biometric Defense against Malicious Generative Editing" (NeurIPS 2026 · similarity 0.84)
- Inadvertent Context Leakage in Language Models (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
