Research map / Multi-agent systems and games / LLM agents

TRACER: Trajectory Risk Aggregation for Critical Episodes in Agentic Reasoning

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TRACER, a trajectory-level uncertainty score for dual-control tool-agent-user dialogues that combines content-aware surprisal, repetition and coherence-gap signals via MAX-composite step risk and tail-focused aggregation, evaluated on τ²-bench failure prediction.

Problem

Estimating uncertainty for LLM agents in multi-turn tool-using interactions with users, where failures come from sparse critical episodes (looping, incoherent tool use, user-agent miscoordination) that token-level uncertainty averages out.

Approach

Score each step with a MAX-composite of content-filtered surprisal, a hybrid semantic-lexical repetition score, and agent/user coherence gaps, then aggregate across the trajectory with a convex mix of top-k tail mean (CVaR-like) and worst-case step risk.

Key results

• TRACER achieves the highest AUROC and AUARC in every evaluated model-domain setting, improving AUROC by 4.7%-37.1% over the strongest baseline.
• On gemini-2.5-pro, TRACER detects 97.3% of retail failures within the first 20% of trajectory vs 66.2% for Semantic Entropy.
• MAX aggregation outperforms additive, multiplicative, and separate-channel variants on AUROC across all 9 model-domain pairs.

Surprising findings

• Multiplicative aggregation of signals can fall to near-chance AUROC (e.g., 0.500 on gemini-2.5-flash retail), indicating dilution of decisive signals.

Limitations

• Tuning of weights (alpha, beta, gamma, k, w) via grid search on held-out episodes; hyperparameter sensitivity not fully characterized.
• Theoretical failure-probability bound is conservative and relies on assumed risk-dominates-hazard and tail-sparsity conditions.
• Evaluated only on three τ²-bench domains and three proprietary models.

Experimental setup

Datasets: τ²-bench airline τ²-bench retail τ²-bench telecom

Benchmarks: τ²-bench

Metrics: AUROC AUARC

Baselines: Normalized entropy SelfConf SemEnt SAUP

Builds on: τ²-bench semantic entropy SAUP Dec-POMDP

Closest papers by meaning

• Position: Uncertainty Quantification Needs Reassessment for Large Language Model Agents (ICML 2025 · similarity 0.87)
• Proper Scoring Rules for Agentic Uncertainty Quantification (NeurIPS 2026 · similarity 0.87)
• Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents (ICML 2026 · similarity 0.86)
• Agentic Confidence Calibration (ICML 2026 · similarity 0.86)
• What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces? (ICML 2026 · similarity 0.85)
• TACT: Mitigating Overthinking and Overacting in Coding Agents via Activation Steering (NeurIPS 2026 · similarity 0.85)