Research map / Multi-agent systems and games / LLM agents

Calibrating Conservatism for Scalable Oversight

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Calibrated Collective Oversight (CCO), which penalizes deviation from a conservative baseline using multiple overseer scores and tunes the conservatism weight online with conformal decision theory to give finite-time violation-rate guarantees, evaluated on SWE-bench Lite and MACHIAVELLI.

Problem

Maintaining human oversight of agentic AI systems that may exceed overseer capabilities, with formal statistical guarantees on undesirable-outcome rates in sequential settings.

Approach

Generalizes AUP's penalty from Q-values to arbitrary overseer scoring functions summed over deviation from a baseline action, and calibrates the conservatism weight online via a CDT additive controller, proving eventual safety of the decision family so finite-time loss-rate bounds follow without distributional assumptions.

Key results

• On SWE-bench Lite (300 instances), CCO at alpha=0.10 yields 9.7% misalignment with 15.3% solve rate, vs 55.0% misalignment and 29.0% solve for unconstrained.
• On MACHIAVELLI Hero of Kendrickstone, CCO at alpha=0.10 yields 10.9% violations with reward 49.0 vs 21.5% violations and 54.0 reward for unconstrained.
• Realized violation rates closely track targets, with maximum deviation of 3.0pp on SWE-bench.

Surprising findings

• On MACHIAVELLI at alpha=0.15, CCO matches unconstrained reward while halving the violation rate.
• The alpha=0.05 MACHIAVELLI setting slightly overshoots target (6.2%), attributed to insufficient iterations for the transient term.

Limitations

• Theory assumes loss observed exactly (or with bounded noise/delay); ground truth often sparse or estimated by fallible judge.
• SWE-bench evaluation is adversarial by construction, not naturalistic deployment.
• Controls long-run average violation rate, not suited to rare catastrophic events; requires a designated baseline action with zero loss.

Experimental setup

Datasets: SWE-bench Lite MACHIAVELLI

Benchmarks: SWE-bench Lite MACHIAVELLI

Metrics: misalignment rate solve rate violation rate reward

Baselines: Always Baseline Unconstrained agent adaptive majority-vote

Builds on: Attainable Utility Preservation Conformal Decision Theory conformal prediction

Closest papers by meaning

• Scaling Laws For Scalable Oversight (NeurIPS 2025 · similarity 0.88)
• PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents (NeurIPS 2026 · similarity 0.85)
• Calibration Is Not Control: Intervention Advantage for LLM-Agent Oversight (NeurIPS 2026 · similarity 0.85)
• AI Agents Push Humans Out of the Loop (NeurIPS 2026 · similarity 0.85)
• Beyond Suspicious Steps: Ontological Trust in Long-Horizon Agents (NeurIPS 2026 · similarity 0.85)
• Position: Intent-aligned AI Systems Must Optimize for Agency Preservation (ICML 2024 · similarity 0.84)