Research map / Multi-agent systems and games / LLM agents

ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ChatInject, which forges native chat-template role tags inside tool outputs (plus a multi-turn variant with synthetic persuasive dialogues) to raise indirect prompt injection success across agents and models.

Problem

Indirect prompt injection against LLM agents has been studied mainly as plain-text injection; the paper asks whether forged chat-template tokens and persuasive multi-turn dialogues embedded in tool outputs can bypass role-based instruction hierarchies.

Approach

Wrap the injected payload (or a synthesized 7-turn persuasive dialogue) in the target model's own chat-template role tags so that the agent treats the tool-output content as higher-priority system/user/assistant turns; also show cross-model transfer via template similarity and mixture-of-templates wrappers.

Key results

• ChatInject raises average ASR from 5.18% to 32.05% on AgentDojo and from 15.13% to 45.90% on InjecAgent; multi-turn ChatInject averages 52.33% on InjecAgent.
• Template-wrapped payloads transfer to closed-source models; e.g. Grok-3 averages 17.9% ASR on InjecAgent with open-source templates.
• Attack ASR tracks embedding similarity between the injected template and the target's native template; existing prompt-based defenses fail to stop it.

Surprising findings

• Multi-turn ChatInject raises ASR yet Utility under attack falls sharply for some models, and GPT-oss utility collapses under nearly any injection.
• Template perturbations (insertion, removal, replacement) that defeat rule-based parsing retain high ASR.

Limitations

• Multi-turn dialogues are synthetically generated by GPT-4.1 and may not reflect real persuasive conversations.
• Closed-source template structures are unknown and are proxied with open-source templates.
• Detector-based defenses cause high false positives and utility loss, and attention analysis is limited.

Experimental setup

Datasets: InjecAgent AgentDojo multi-turn jailbreak dataset (adapted)

Benchmarks: InjecAgent AgentDojo

Metrics: Attack Success Rate Utility under Attack benign utility embedding cosine similarity

Baselines: Default InjecPrompt Default Multi-turn PI detector Lakera Guard Instructional Prevention Data Delimiters User Instruction Repetition

Builds on: InjecAgent AgentDojo InjecPrompt ChatBug

Closest papers by meaning

• ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates (AAAI 2025 · similarity 0.91)
• Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game (ICLR 2024 · similarity 0.88)
• WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models (NeurIPS 2024 · similarity 0.88)
• An LLM can Fool Itself: A Prompt-Based Adversarial Attack (ICLR 2024 · similarity 0.87)
• Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards (ICML 2025 · similarity 0.87)
• Memory Injection Attacks on LLM Agents via Query-Only Interaction (NeurIPS 2025 · similarity 0.87)