Research map / Multi-agent systems and games / LLM agents

Watermarking LLM Agent Trajectories

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AgentWmk, a behavior-level watermark that injects hook actions into LLM agent trajectory datasets, activated by a secret key, and detects dataset use by black-box hypothesis testing of hook action rates.

Problem

Copyright protection and provenance tracing for LLM agent trajectory datasets, which are small (1-2K samples), have interleaved action-observation structure, and are costly to create; existing dataset watermarks fail on them.

Approach

Insert semantically harmless hook actions (drawn from the dataset's existing action space) at action boundaries, paired with a secret activation key in the user prompt; a model trained on this data emits hook actions at a higher rate when the key is present, and a paired one-sided t-test against a sham key verifies the watermark.

Key results

• Average detection AUC of 94.3 on Qwen-2.5-Coder-7B across MATH, SimpleQA, and SWE-Smith with one prompt and eight queries; Standalone 97.8 and Contextual 90.8 vs CodeMark 55.5.
• Pass@1 changes are negligible; e.g., Qwen-2.5-Coder-7B MATH 75.3 without watermark vs 75.3 Standalone and 75.3 Contextual.
• Under DeCoMa filtering, Standalone AUC stays at 97.3 (MATH) and 98.8 (SimpleQA) while CodeMark drops to about 50-54.

Surprising findings

• CodeMark is near chance (AUC about 50-57) on small agent-trajectory datasets, indicating token-level watermarks are ineffective in this low-data regime.
• Contextual watermarks degrade substantially under paraphrasing on SWE-Smith (AUC 94.2 to 56.0).

Limitations

• Contextual watermarks degrade under paraphrasing on long SWE-Smith trajectories and on smaller models (3B) or non-code models (Llama-3.1-8B).
• Watermark detection on SWE-Smith is weaker, with AUC around 52-59 at 3B scale.
• Accuracy of adapted backdoor baselines on SWE-Smith was not evaluated (bash adaptation left to future work).

Experimental setup

Datasets: MATH SimpleQA SWE-Smith

Benchmarks: MATH SimpleQA SWE-Bench Lite Smolagents-Benchmark-v1

Metrics: AUC TPR@1%FPR TPR@5%FPR Pass@1 average turns average output tokens paired t-statistic precision recall F1

Baselines: CodeMark AutoPoison DeadCode StyleTransfer

Builds on: CodeMark AutoPoison DeadCode DeCoMa radioactivity

Closest papers by meaning

• Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces (NeurIPS 2026 · similarity 0.83)
• When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents (ICML 2026 · similarity 0.81)
• Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents (ICML 2026 · similarity 0.81)
• Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark (AAAI 2026 · similarity 0.81)
• Watermark Stealing in Large Language Models (ICML 2024 · similarity 0.81)
• Can Watermarked LLMs be Identified by Users via Crafted Prompts? (ICLR 2025 · similarity 0.81)