AI Paper Atlas

Research map / Multi-agent systems and games / LLM agents

UMAS: System-Level Uncertainty Quantification for Multi-Agent LLM Systems

Hanwen Li, Jinhao Duan, Xiaoshuang Shi, Yue Zhang, Tianlong Chen, Kaidi Xu, Chenxi Yuan

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UMAS, a system-level uncertainty quantification method for multi-agent LLM systems that exploits MAS structure while keeping inference cost neutral.

Problem

Multi-agent LLM systems are prone to hallucinations and erroneous outputs, but existing uncertainty quantification methods are designed for single-agent settings and do not exploit multi-agent structure.

Approach

Designs uncertainty quantification that operates at the system level by using the structure of multi-agent interactions, with cost-neutral inference (no extra inference calls, per the title and abstract framing).

Measured effects reported in the paper

What was changedWhereMeasuredEffect
aggregate uncertainty over the whole MAS trajectory instead of the terminal output (UMAS-Fix vs best baseline) (ablation)MSP-based UQ · Average AUROC, Llama 4 ScoutAUROC (%): 63.00 -> 70.32Improved
aggregate uncertainty over the whole MAS trajectory instead of the terminal output (UMAS-Fix vs best baseline) (ablation)MSP-based UQ · Average AUROC, GPT-4o miniAUROC (%): 64.86 -> 71.49Improved
add credibility-aware confidence updating on top of aggregation (ablation)UMAS-Fix · Average AUROC, Llama 4 ScoutAUROC (%): 70.32 -> 74.93Improved
add credibility-aware confidence updating on top of aggregation (ablation)UMAS-Fix · Average AUROC, GPT-4o miniAUROC (%): 71.49 -> 74.02Improved
credibility-aware updating helps on Debate GSM-Hard (ablation)UMAS-Fix · Debate GSM-Hard, Llama 4 ScoutAUROC (%): 64.25 -> 72.69Improved
credibility-aware updating on DyLAN MBPP (ablation)UMAS-Fix · DyLAN MBPP, Llama 4 ScoutAUROC (%): 51.93 -> 56.98Improved
credibility-aware updating on Debate MMLU-Pro with GPT-4o mini (ablation)UMAS-Fix · Debate MMLU-Pro, GPT-4o miniAUROC (%): 77.42 -> 74.40Hurt
use PPL or PE as node-level confidence instead of MSP (ablation)UMAS · Debate, Llama 4 Scout (MMLU-Pro, AQUA, MedMCQA, MBPP)AUROC (%): PPL MMLU-Pro 67.67 -> 77.98Improved
apply UMAS on PPL for MBPP (ablation)UMAS with PPL confidence · Debate MBPP, Llama 4 ScoutAUROC (%): 60.03 -> 59.87No clear effect
select the lowest-uncertainty trajectory out of 5 using UMASDebate (GPT-4o mini) · AQUA-RATaccuracy (%): 83.7 -> 96.3Improved
select the lowest-uncertainty trajectory out of 5 using UMASDyLAN (GPT-4o mini) · MMLU-Proaccuracy (%): 71.0 -> 76.5Improved
use single-trajectory logit-based UQ instead of multi-sampling consistency UQUMAS · Debate MMLU-Pro, Llama 4 Scoutinference latency (s): 127.32 -> 84.17Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy