Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes

Proper Scoring Rules for Agentic Uncertainty Quantification

NeurIPS 2026 · Probabilistic and scientific modeling › Uncertainty and Gaussian processes · approach: Regression uncertainty quantification · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the Trajectory Proper Score (TPS), a family of strictly proper weighted trajectory scoring rules for prefix-conditioned success probabilities, with censoring extensions, and shows common agentic UQ evaluators (AUROC, T-ECE, scalarized Brier) fail to elicit the full probability trace.

Problem

Agentic uncertainty evaluations use AUROC, ECE, and scalarized Brier, which do not strictly elicit the full prefix-conditioned success-probability trace, and they mishandle administratively censored trajectories.

Approach

Lift strictly proper binary scores (beta family) to trajectories via positive-weighted sums over prefixes, extend to right-censored traces by conditional projection onto the observable prefix, and prove that T-ECE and scalarized Brier fail to elicit the full trace.

Key results

• On Tau2-Bench (n=201), Platt recalibration changes AUROC by only -0.010 (about 0.3 SE) but improves TPS-log by 5.67 nats (about 43 SE).
• On WebShop (n=308, 47.08% administrative censoring), censored-aware evaluation shifts the verbal-confidence TPS-log by +0.159 nats (95% CI [+0.133, +0.188]) relative to complete-only evaluation.
• Theorems show T-ECE can tie a constant non-truthful forecast with the truthful forecast, and scalarized last/average/min Brier fail strict propriety for the trace.

Surprising findings

• Recalibrating verbal confidence leaves AUROC nearly unchanged while TPS-log shifts by over 5 nats, exposing probability-scale mis-scaling invisible to rank metrics.
• Censored-aware scoring changes the verdict relative to complete-only evaluation, since censored trajectories are structurally longer and harder.

Limitations

• Censored extension requires non-informative administrative censoring; parse-error truncations and adaptive stops violate it.
• Targets binary terminal outcomes only; partial credit and multi-objective settings are not covered.
• Empirical results use a single model and one ReAct harness.

Experimental setup

Datasets: StrategyQA Tau2-Bench HotpotQA WebShop

Benchmarks: Tau2-Bench StrategyQA HotpotQA WebShop

Metrics: Trajectory Proper Score (log, Brier, Beta(2,4)) AUROC AUPRC AURC Trajectory ECE

Baselines: AUROC AUPRC risk-coverage/AURC Trajectory ECE scalarized Trajectory Brier base-rate reference

Builds on: Gneiting and Raftery proper scoring rules Schervish threshold-mixture representation prequential forecasting Reliability-Resolution-Uncertainty decomposition

Closest papers by meaning

• Agentic Confidence Calibration (ICML 2026 · similarity 0.88)
• TRACER: Trajectory Risk Aggregation for Critical Episodes in Agentic Reasoning (ICML 2026 · similarity 0.87)
• Offline Preference-Based Trajectory Evaluation (NeurIPS 2026 · similarity 0.87)
• Unified Uncertainty-Aware Diffusion for Multi-Agent Trajectory Modeling (CVPR 2025 · similarity 0.86)
• Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability (NeurIPS 2026 · similarity 0.86)
• ExComm: Exploration-Stage Communication for Error-Resilient Agentic Test-Time Scaling (NeurIPS 2026 · similarity 0.85)