Research map / Multi-agent systems and games / LLM agents

When Planning Fails Despite Correct Execution: On Epistemic Calibration for LLM-Based Multi-Agent Systems

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EPC-AW, a planning-time workflow that selects plans whose feasibility evaluations are stable across heterogeneously informed agents (IPS) and stores cross-round discrepancies as memory constraints (CESR), improving multi-agent QA/search accuracy by about 9.75% on average.

Problem

LLM-based multi-agent systems fail even when every action executes correctly, because agents miscalibrate feasibility of plans under partial and evolving information; this miscalibration is latent and dynamic.

Approach

Instead of verifying plan feasibility directly, select plans whose evaluations remain consistent across agents with different private memories (cross-agent information-consistency score), and convert planner-vs-consistency-selected plan divergences into persistent natural-language epistemic constraints in the planner's memory.

Key results

• EPC-AW improves average accuracy by 9.75% over No-Repair on six benchmarks, and by 4.64% over Rollback (Qwen3-Coder-30B backbone).
• On Bamboogle EPC-AW reaches 58.13% vs 52.27% for Rollback and 48.27% for No-Repair.
• Across backbones, EPC-AW gains 11.18% on Qwen3-14B and 11.13% on DeepSeek-R1-32B over No-Repair.

Surprising findings

• IPS alone degrades performance on 2Wiki (36.83 vs 39.00 for no-repair-style baseline in ablation), suggesting overly conservative planning without CESR.
• Mean-score aggregation slightly outperforms IPS on 2Wiki (52.33 vs 51.83).

Limitations

• Consistency score is a heuristic; the paper gives no theoretical guarantee that consistency tracks feasibility.
• Experiments use a single backbone family in the main results and small evaluation subsets (e.g., 100-300 samples) with three runs.
• Search pipeline was modified (local Chrome-based Google search and HuggingFace filtering), which may limit comparability with AgentFlow.

Experimental setup

Datasets: Bamboogle 2Wiki HotpotQA Musique GAIA MedQA

Benchmarks: Bamboogle 2Wiki HotpotQA Musique GAIA MedQA

Metrics: accuracy absolute improvement

Baselines: No-Repair (AgentFlow) Retry Rollback Mean-Score Aggregation

Builds on: AgentFlow MAST COCO

Closest papers by meaning

• Towards Epistemic-Doxastic Planning with Observation and Revision (AAAI 2024 · similarity 0.87)
• Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability (NeurIPS 2026 · similarity 0.86)
• ExComm: Exploration-Stage Communication for Error-Resilient Agentic Test-Time Scaling (NeurIPS 2026 · similarity 0.85)
• Agentic Confidence Calibration (ICML 2026 · similarity 0.85)
• MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems (ICML 2026 · similarity 0.84)
• Towards a Science of AI Agent Reliability (ICML 2026 · similarity 0.84)