Research map / Multi-agent systems and games / LLM agents
Belief-Calibrated Multi-Agent Consensus Seeking for Complex NLP Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BCCS, a belief-calibrated multi-agent consensus framework that judges consensus using agent beliefs (answer generation probabilities) and selects collaborators and leaders via conflict scores, evaluated on MATH and MMLU with Qwen2.5-7B-Instruct agents.
Problem
Existing multi-agent consensus-seeking relies on voting/answer agreement, ignoring internal belief contradictions, and has agents indiscriminately collaborate with all others, which hinders stable consensus on complex NLP tasks.
Approach
Calibrate consensus judgment with agent belief (generation probability) into full/partial/no consensus states; for partial consensus, assign each agent the highest-belief supportive or conflicting collaborator via a macro/micro conflict score; for no consensus, pick the highest-belief agents of each opinion group as leaders. Theoretical convergence results under opinion-dynamics assumptions suppor
Key results
- On MATH, BCCS reaches 80.60 average accuracy vs 78.93 for the strongest baseline CMD (about 1.67 points better on average, up to 2.23 on hard subjects).
- On MMLU, BCCS reaches 78.47 average accuracy vs 76.47 for PARSE (about 2.00 points on average, up to 3.95 on the Other/Humanities subsets).
- Ablations show removing BCCJ, CA, or LS lowers MATH average accuracy by 2.60, 2.00, and 2.47 points respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use BCCS instead of the strongest baseline, CMD | multi-agent reasoning · MATH overall | accuracy: 78.93\textpm0.53 -> 80.60\textpm0.23 | Helped |
| use BCCS instead of the strongest baseline, PARSE | multi-agent reasoning · MMLU overall | accuracy: 76.47\textpm0.48 -> 78.47\textpm1.22 | Helped |
| add collaborator assignment (ablation) | BCCS · MATH overall | accuracy: 78.60\textpm0.12 -> 80.60\textpm0.23 | Helped |
| include conflicting agents rather than using supportive opinions only (ablation) | BCCS · MATH overall | accuracy: 79.00\textpm0.23 -> 80.60\textpm0.23 | Helped |
| add leader selection (ablation) | BCCS · MATH overall | accuracy: 78.13\textpm0.48 -> 80.60\textpm0.23 | Helped |
| select leaders by belief instead of randomly (ablation) | BCCS · MATH overall | accuracy: 78.33\textpm0.66 -> 80.60\textpm0.23 | Helped |
| use belief-calibrated consensus judgment instead of Byzantine Consensus (ablation) | BCCS · MATH overall | accuracy: 78.00\textpm0.53 -> 80.60\textpm0.23 | Helped |
| add collaborator assignment (ablation) | BCCS · MMLU overall | accuracy: 76.67\textpm0.29 -> 78.47\textpm1.22 | Helped |
| include conflicting agents rather than using supportive opinions only (ablation) | BCCS · MMLU overall | accuracy: 77.20\textpm0.81 -> 78.47\textpm1.22 | Helped |
| add leader selection (ablation) | BCCS · MMLU overall | accuracy: 76.73\textpm1.20 -> 78.47\textpm1.22 | Helped |
| select leaders by belief instead of randomly (ablation) | BCCS · MMLU overall | accuracy: 76.60\textpm1.06 -> 78.47\textpm1.22 | Helped |
| use belief-calibrated consensus judgment instead of Byzantine Consensus (ablation) | BCCS · MMLU overall | accuracy: 77.13\textpm1.01 -> 78.47\textpm1.22 | Helped |
Surprising findings
- Collaborating only with supportive agents (no conflicting agents) lowers accuracy, with success consensus level peaking at intermediate supportive ratios.
Limitations
- Not evaluated in embodied intelligence environments.
- Requires extra time to compute conflict scores for collaborator selection.
- Belief is approximated by LLM output (generation) probability, which may not reflect true uncertainty.
Experimental setup
Datasets: MATH MMLU FOLIO CommonsenseQA
Benchmarks: MATH MMLU FOLIO CommonsenseQA
Metrics: accuracy standard error of the mean consensus level success consensus rate tokens per case
Baselines: CoT Reflection CoT-SC EoT GroupDebate MAD PARSE DyLAN CMD
Builds on: Byzantine consensus chain-of-thought CMD PARSE MAD
Closest papers by meaning
- Multi-Agent Debate for LLM Judges with Adaptive Stability Detection (NeurIPS 2025 · similarity 0.88)
- Efficient Multi-Agent Reasoning via Confidence-Guided Adaptive Debate (ICML 2026 · similarity 0.87)
- Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs (ICML 2026 · similarity 0.86)
- From Debate to Equilibrium: Belief‑Driven Multi‑Agent LLM Reasoning via Bayesian Nash Equilibrium (ICML 2025 · similarity 0.86)
- Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models? (NeurIPS 2025 · similarity 0.86)
- AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.