Research map / Multi-agent systems and games / LLM agents

ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

ICLR 2024 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent collaboration and safety · 55 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ChatEval, a multi-agent LLM debate framework in which referees with diverse personas discuss and judge text quality, evaluated on FairEval and Topical-Chat.

Problem

Single-agent LLM evaluators fall short of human-level text evaluation quality, and human evaluation involves multiple annotators collaborating, which single LLM judges do not mimic.

Approach

Treat each LLM as a referee agent with a distinct persona that debates with others under configurable communication strategies (one-by-one, simultaneous-talk, simultaneous-talk-with-summarizer), with final verdicts from majority vote or averaged scores; diverse roles are shown to be essential.

Key results

• On FairEval, GPT-4 multi-agent reaches 63.8% accuracy vs 61.3% single-agent and 62.5% for FairEval's MEC+BPC; ChatGPT multi-agent reaches 60.0% vs 53.8% single-agent.
• On Topical-Chat with GPT-4, multi-agent averages Spearman 0.684 and Kendall-Tau 0.632 versus 0.658 and 0.611 single-agent and 0.588 and 0.575 for G-EVAL-4.
• Replacing diverse role prompts with an identical role prompt drops ChatGPT multi-agent accuracy from 60.0% back to 53.8% (single-agent level).

Measured effects reported in the paper

Surprising findings

• Using the same role description for all agents yields no improvement over single-agent evaluation, showing persona diversity is necessary.
• Increasing discussion turns does not give a consistent gain and can lead to stagnation or degradation, attributed to growing context length.

Limitations

• Only homogeneous groups of GPT models were studied; heterogeneous groups are left to future work.
• Gains on FairEval are small (e.g., 2.5 points for GPT-4) and results are on small benchmarks (80 questions, 60 dialogue contexts).
• Discussion turns beyond a few show no benefit and increase context length.

Experimental setup

Datasets: FairEval Topical-Chat

Benchmarks: FairEval Topical-Chat

Metrics: Accuracy Kappa Spearman correlation Kendall-Tau correlation

Baselines: Single-Agent FairEval (MEC+BPC) G-EVAL-3.5 G-EVAL-4 ROUGE-L BLEU-4 BERTScore

Builds on: G-EVAL FairEval AgentVerse

Closest papers by meaning

• IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering (NeurIPS 2024 · similarity 0.90)
• Unbiased Evaluation of Large Language Models from a Causal Perspective (ICML 2025 · similarity 0.89)
• Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs (ICML 2024 · similarity 0.89)
• Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis (ICLR 2026 · similarity 0.89)
• ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluation (NeurIPS 2024 · similarity 0.89)
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.88)