Research map / Multi-agent systems and games / LLM agents

Calibration Is Not Control: Intervention Advantage for LLM-Agent Oversight

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Argues that LLM-agent oversight should score intervention advantage rather than failure risk, and shows via prefix branching on ALFWorld and other benchmarks that intervention-trained controllers cut regret and improve closed-loop success with fewer handoffs.

Problem

Runtime oversight of LLM agents typically thresholds a calibrated failure score to decide when to intervene, but states with equal failure risk can differ in whether intervention helps, so risk scores may lose control-relevant information.

Approach

Formalizes sufficiency of a summary for intervention decisions (intervention advantage tau = Q_i - Q_continue), and uses prefix branching (replaying prefixes and executing all actions) to supervise controllers with intervention-outcome targets while keeping features, estimator, and router fixed.

Key results

• On ALFWorld, regret falls from 0.506 for failure-triggered routing to 0.110 for the prefix-feature controller (reduction 0.396, 95% CI [0.310, 0.488]).
• Changing only the supervision target from failure label to intervention utility lowers regret from 0.513 to 0.094 with features, estimator, and router fixed.
• In closed loop on 300 unseen ALFWorld episodes with a GPT-5.4 handoff, the prefix-feature controller hands off on 35% vs 48% and succeeds on 45% vs 37% for failure-triggered routing.

Surprising findings

• Platt scaling cuts the failure score's calibration error from 0.463 to 0.006 yet leaves control regret essentially unchanged.
• Always handing off succeeds on 40% of episodes, less often than the learned controller handing off on about a third of episodes.

Limitations

• Strong results concentrated on ALFWorld; gains small on GSM8K and HotpotQA reasoning benchmarks.
• Online gain not attributed between better stopping and better handoffs since no learned continue-or-quit controller or fixed handoff budget was used.
• Branching is a development-time cost (about 2x suffix cost), and realized branch utilities are single samples.

Experimental setup

Datasets: ALFWorld ScienceWorld GSM8K HotpotQA

Benchmarks: ALFWorld ScienceWorld GSM8K HotpotQA

Metrics: regret utility success rate handoff rate AUROC calibration error

Baselines: failure-triggered routing random forest on failure score continuation utility target compact summary controller always hand off continue only quit at first decision

Builds on: learning to defer Platt scaling isotonic regression

Closest papers by meaning

• Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias (NeurIPS 2026 · similarity 0.88)
• Building a Foundational Guardrail for General Agentic Systems via Synthetic Data (ICLR 2026 · similarity 0.87)
• PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization (NeurIPS 2026 · similarity 0.86)
• Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors (NeurIPS 2026 · similarity 0.86)
• Conformal Selective Acting: Anytime-Valid Risk Control for RLVR-Trained LLMs (NeurIPS 2026 · similarity 0.86)
• AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems (NeurIPS 2026 · similarity 0.86)