Research map / Multi-agent systems and games / LLM agents

Sequential Probabilistic Uncertainty Estimation for Parallel Multi-Agent Reasoning Systems

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SAUCE, a training-free uncertainty estimator for parallel multi-agent debate/DyLAN systems that runs a Kalman-style filter over round-level modal-answer agreement weighted by token entropy.

Problem

Uncertainty estimation for parallel reasoning multi-agent systems, where reliability depends on how agent consensus evolves across rounds rather than on the final output alone.

Approach

Treats round-level modal-answer agreement as a noisy observation of a latent system belief, with per-round generation entropy as observation noise, and applies a linear-Gaussian filtering recursion; the uncertainty score is the negated temporal average of the posterior mean.

Key results

• On open-source Qwen3-4B, SAUCE averages 73.50 AUROC and 85.47 AUARC across benchmarks and protocols, above the best baseline LL at 66.79 AUROC.
• On Llama-3.1-8B-Instruct, SAUCE reaches 76.84 average AUROC versus 67.72 for UDPO.
• On closed-source Gemini-2.5-Flash-Lite, SAUCE averages 75.77 AUROC versus 65.29 for UDPO; SAUCE has the lowest Brier-Platt on four of five backbones.

Surprising findings

• Single-round PE and LL scores often land near or below chance (e.g., Llama-3.1-8B AUROC around 35-65) under MAS protocols.
• A static precision-weighted combination of agreement and generation entropy performs about the same as the agreement-only round mean, so gains come from the sequential update.

Limitations

• Applies only to parallel reasoning MAS with comparable per-round answers; not directly applicable to heterogeneous-role or tool-using agentic systems.
• Uses simple agreement (modal-answer share) and token-entropy signals; principled alternatives are left for future work.
• Hyperparameters tuned on a single calibration cell (Qwen3-4B / MMLU-Pro) and transferred; per-cell tuning benefit not bounded.

Experimental setup

Datasets: MATH-500 MMLU-Pro BBH OlympiadBench BBEH-mini

Benchmarks: MATH-500 MMLU-Pro BBH OlympiadBench BBEH-mini

Metrics: AUROC AUARC Brier score after Platt scaling

Baselines: PE LL UDPO MATU

Builds on: Debate DyLAN MATU UDPO

Closest papers by meaning

• Epistemic Gain, Aleatoric Cost: Uncertainty Decomposition in Multi-Agent Debate for Math Reasoning (ICML 2026 · similarity 0.88)
• From Debate to Equilibrium: Belief‑Driven Multi‑Agent LLM Reasoning via Bayesian Nash Equilibrium (ICML 2025 · similarity 0.87)
• Position: Uncertainty Quantification Needs Reassessment for Large Language Model Agents (ICML 2025 · similarity 0.87)
• Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs (ICML 2026 · similarity 0.87)
• The Value of Variance: Mitigating Debate Collapse in Multi-Agent Systems via Uncertainty-Driven Policy Optimization (ICML 2026 · similarity 0.86)
• MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models (ICML 2024 · similarity 0.86)