Research map / Multi-agent systems and games / LLM agents

Constitutional Black-Box Monitoring for Scheming in LLM Agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent collaboration and safety · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds constitutional black-box scheming monitors by optimizing prompted frontier LLM classifiers on 2,000 synthetic trajectories (STRIDE and Gloom) and tests them on 7,500 semi-realistic ControlArena trajectories, finding performance saturates at simple prompt sweeps.

Problem

Detecting scheming (covert pursuit of misaligned goals) in LLM agents from externally observable behavior, where real scheming examples are scarce and the low false-positive regime matters for deployment.

Approach

Generate synthetic scheming and benign agent trajectories from a natural-language scheming constitution using two pipelines (STRIDE single-call generation with discriminator refinement, and Gloom agent-environment simulation), optimize prompted monitors on them with prompt sweeps, human refinement, and DSPy automated optimization, and evaluate transfer to executable ControlArena environments with

Key results

• Grid-searched monitors reach test pAUROC of 0.853 (Gloom-optimized) and 0.866 (STRIDE-optimized) on 7,500 ControlArena trajectories.
• Heavier optimization (human refinement, GEPA, MIPROv2) yields no significant test gains over prompt sweep, except human refinement on STRIDE.
• Monitors trained on 1,000 in-distribution ControlArena trajectories reach 0.81 pAUROC, comparable to synthetic-trained monitors, suggesting the ceiling is not solely due to synthetic-to-real shift.

Surprising findings

• Aggressive optimization on synthetic data overfits: additional prompt refinement improves training performance but stagnates or degrades test performance.
• Monitors optimized on synthetic data match monitors optimized on in-distribution ControlArena data, implying the plateau is not only distribution shift.

Limitations

• Evaluation uses semi-realistic sandboxes with side tasks injected via system prompts rather than naturally arising scheming.
• Kimi K2.6 fine-tuning pilot is limited to a single base model, single LoRA configuration, and no exhaustive hyperparameter search.
• Frontier fine-tuning APIs rejected the scheming-themed training data, precluding like-for-like fine-tuning comparison.

Experimental setup

Datasets: STRIDE synthetic trajectories Gloom synthetic trajectories ControlArena trajectories

Benchmarks: ControlArena

Metrics: partial AUROC (FPR 0-0.2) AUROC

Baselines: minimal baseline prompt prompt sweep GEPA MIPROv2 supervised fine-tuning

Builds on: Constitutional Classifiers Bloom DSPy GEPA MIPROv2

Closest papers by meaning

• SchemeArena: Factorized Stress Testing of Scheming in LLM Agents (NeurIPS 2026 · similarity 0.90)
• Evaluating and Understanding Scheming Propensity in LLM Agents (NeurIPS 2026 · similarity 0.89)
• Agent-ToM: Learning to Monitor Autonomous LLM Agents via Theory-of-Mind Reasoning (NeurIPS 2026 · similarity 0.89)
• Monitoring Decomposition Attacks with Lightweight Sequential Monitors (ICLR 2026 · similarity 0.88)
• Building a Foundational Guardrail for General Agentic Systems via Synthetic Data (ICLR 2026 · similarity 0.88)
• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition (ICML 2026 · similarity 0.87)