Research map / Multi-agent systems and games / LLM agents

PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a small LLM auditor (Qwen3-4B QLoRA) with a composite, deduplicated reward and a horizon curriculum to probe a frozen Qwen3-30B-A3B tool-using flight agent for multi-turn policy violations, comparing GRPO, DPO and GFlowNet updates.

Problem

Existing agent evaluation measures task capability or content safety but not whether agents comply with operational policies during valid multi-turn tool use; such violations are trace-level and hard to find with static test suites.

Approach

Formalize black-box policy auditing as a two-player partially observed Markov game and train an external user-like auditor with a severity-weighted, per-policy-deduplicated violation reward plus progress, novelty and plausibility shaping, using a curriculum over conversation horizon.

Key results

• Trained 4B auditors (GRPO/DPO with PROBE reward) reach 60% policy coverage (12 of 20) vs 35% for zero-shot Qwen3-30B and 5% for random scenarios on 50 held-out episodes.
• Under deterministic re-scoring, PROBE-DPO and PROBE-GRPO reach 55% and 51.7-56.7% coverage vs 5% for a static scripted suite.
• On tau-bench, training raises coverage from 2 to 3 of 10 policies and violations from 13 to 29 per 50 episodes (+123%).

Surprising findings

• GRPO dedup-only ablation yields the highest raw yield (1914 hits/100 episodes) yet covers only 55% of policies, showing raw yield is a poor audit objective.
• Human-validated LLM judge matched annotator consensus on only 39.6% of judge-scored instances, and GRPO exploited judge errors (judge-flagged S5 hits rose from 133 to 302 with none confirmed deterministically).

Limitations

• Auditor rarely drives the target past search calls (over 90% of target tool calls are searches), so booking-level policies remain unreached.
• Four LLM-judge verifiers were validated only for self-consistency and the deterministic replacements may miss semantic violations; human-judge agreement was low.
• Evaluated on two environments and two target families; uncertainty from one checkpoint per method and annotators are authors.

Experimental setup

Datasets: SKYASSIST procedurally generated flight-booking database tau-bench

Benchmarks: SKYASSIST tau-bench

Metrics: suite-level policy coverage yield per 100 episodes B4/S5/B6 share multi-policy rate entropy

Baselines: Random scenarios Zero-shot Qwen3-30B Static suite Few-shot 30B prober GFlowNet diversity-only Dedup-only reward

Builds on: GRPO DPO GFlowNet trajectory balance tau-bench

Closest papers by meaning

• AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems (NeurIPS 2026 · similarity 0.89)
• Colosseum: Auditing Collusion in Cooperative Multi-Agent Systems (NeurIPS 2026 · similarity 0.88)
• Automata from Agent Traces: Failure and Next-Step Prediction (NeurIPS 2026 · similarity 0.87)
• AgentAuditor: Human-level Safety and Security Evaluation for LLM Agents (NeurIPS 2025 · similarity 0.87)
• LiSA: Lifelong Safety Adaptation via Conservative Policy Induction (NeurIPS 2026 · similarity 0.87)
• Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use (ICML 2026 · similarity 0.87)