Research map / Multi-agent systems and games / LLM agents
UMAS: System-Level Uncertainty Quantification for Multi-Agent LLM Systems
Hanwen Li, Jinhao Duan, Xiaoshuang Shi, Yue Zhang, Tianlong Chen, Kaidi Xu, Chenxi Yuan
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UMAS, a system-level uncertainty quantification method for multi-agent LLM systems that exploits MAS structure while keeping inference cost neutral.
Problem
Multi-agent LLM systems are prone to hallucinations and erroneous outputs, but existing uncertainty quantification methods are designed for single-agent settings and do not exploit multi-agent structure.
Approach
Designs uncertainty quantification that operates at the system level by using the structure of multi-agent interactions, with cost-neutral inference (no extra inference calls, per the title and abstract framing).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| aggregate uncertainty over the whole MAS trajectory instead of the terminal output (UMAS-Fix vs best baseline) (ablation) | MSP-based UQ · Average AUROC, Llama 4 Scout | AUROC (%): 63.00 -> 70.32 | Improved |
| aggregate uncertainty over the whole MAS trajectory instead of the terminal output (UMAS-Fix vs best baseline) (ablation) | MSP-based UQ · Average AUROC, GPT-4o mini | AUROC (%): 64.86 -> 71.49 | Improved |
| add credibility-aware confidence updating on top of aggregation (ablation) | UMAS-Fix · Average AUROC, Llama 4 Scout | AUROC (%): 70.32 -> 74.93 | Improved |
| add credibility-aware confidence updating on top of aggregation (ablation) | UMAS-Fix · Average AUROC, GPT-4o mini | AUROC (%): 71.49 -> 74.02 | Improved |
| credibility-aware updating helps on Debate GSM-Hard (ablation) | UMAS-Fix · Debate GSM-Hard, Llama 4 Scout | AUROC (%): 64.25 -> 72.69 | Improved |
| credibility-aware updating on DyLAN MBPP (ablation) | UMAS-Fix · DyLAN MBPP, Llama 4 Scout | AUROC (%): 51.93 -> 56.98 | Improved |
| credibility-aware updating on Debate MMLU-Pro with GPT-4o mini (ablation) | UMAS-Fix · Debate MMLU-Pro, GPT-4o mini | AUROC (%): 77.42 -> 74.40 | Hurt |
| use PPL or PE as node-level confidence instead of MSP (ablation) | UMAS · Debate, Llama 4 Scout (MMLU-Pro, AQUA, MedMCQA, MBPP) | AUROC (%): PPL MMLU-Pro 67.67 -> 77.98 | Improved |
| apply UMAS on PPL for MBPP (ablation) | UMAS with PPL confidence · Debate MBPP, Llama 4 Scout | AUROC (%): 60.03 -> 59.87 | No clear effect |
| select the lowest-uncertainty trajectory out of 5 using UMAS | Debate (GPT-4o mini) · AQUA-RAT | accuracy (%): 83.7 -> 96.3 | Improved |
| select the lowest-uncertainty trajectory out of 5 using UMAS | DyLAN (GPT-4o mini) · MMLU-Pro | accuracy (%): 71.0 -> 76.5 | Improved |
| use single-trajectory logit-based UQ instead of multi-sampling consistency UQ | UMAS · Debate MMLU-Pro, Llama 4 Scout | inference latency (s): 127.32 -> 84.17 | Improved |
Closest papers by meaning
- Sequential Probabilistic Uncertainty Estimation for Parallel Multi-Agent Reasoning Systems (NeurIPS 2026 · similarity 0.91)
- Position: Uncertainty Quantification Needs Reassessment for Large Language Model Agents (ICML 2025 · similarity 0.89)
- The Value of Variance: Mitigating Debate Collapse in Multi-Agent Systems via Uncertainty-Driven Policy Optimization (ICML 2026 · similarity 0.88)
- AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition (ICML 2026 · similarity 0.87)
- MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems (ICML 2026 · similarity 0.87)
- Uncertainty-Aware Clarification in LLM Agents with Information Gain (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
