Research map / Multi-agent systems and games / LLM agents

GLEAN: Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verification

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GLEAN, a guideline-grounded verifier that accumulates per-step guideline-alignment logits along an agent trajectory, calibrates them with Bayesian logistic regression, and triggers extra evidence collection on uncertain cases for clinical diagnosis agents.

Problem

Verifying correctness of LLM agent decisions in high-stakes settings such as clinical diagnosis, where existing verifiers lack domain knowledge and produce poorly calibrated confidence.

Approach

Treat verification as sequential evidence accumulation: an LLM judge scores each step against expert clinical guidelines (YES/NO token probability), the logits are discount-summed into a surrogate evidence signal, and a Bayesian logistic calibrator maps it to correctness probability; high-entropy cases trigger guideline expansion and differential checks against competing diagnoses.

Key results

• With active verification, GLEAN reaches AUROC 0.9756 (diverticulitis) and 0.9862 (Qwen3-30B backbone) versus best baseline ~0.85 on Qwen2.5-7B, per the paper's table.
• Best-of-N accuracy for Qwen3-30B rises from 58.94% to 78.3% at N=16 with GLEAN, versus 70.0% for self-consistency.
• Expert study with three clinicians rated GLEAN's usefulness 4.67/5 with Cohen's kappa 0.78 on error identification.

Surprising findings

• Model-based ratings without guideline grounding show no statistically significant separation between correct and incorrect prefixes.

Limitations

• Evaluated only on clinical diagnosis with three diseases from MIMIC-IV, so generalization to other high-stakes domains is untested.
• Calibration can be partially agent-specific; cross-agent calibration degrades for Qwen3-30B verifying the other agent.
• Relies on guideline quality and coverage; guidelines may be incomplete or outdated.

Experimental setup

Datasets: MIMIC-IV epfl-llm/guidelines

Benchmarks: MIMIC-IV diverticulitis MIMIC-IV cholecystitis MIMIC-IV pancreatitis

Metrics: AUROC Risk@0.5 ECE Brier score Best-of-N accuracy

Baselines: P(TRUE) LLM-as-a-Judge Self-Consistency Semantic Entropy Self-Verification RAG-Augmented rating Med-PRM ORM

Builds on: ReAct Med-PRM self-consistency semantic entropy

Closest papers by meaning

• MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow (ICLR 2026 · similarity 0.84)
• PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (NeurIPS 2026 · similarity 0.83)
• Can AI Agents Synthesize Scientific Conclusions? (NeurIPS 2026 · similarity 0.83)
• Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing (NeurIPS 2026 · similarity 0.83)
• MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making (NeurIPS 2024 · similarity 0.82)
• DDx-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs (NeurIPS 2026 · similarity 0.82)