Research map / Multi-agent systems and games / LLM agents

DEBATE: A Large-Scale Benchmark for Evaluating Opinion Dynamics in Role-Playing LLM Agents

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DEBATE, a large-scale human group-deliberation benchmark (2,788 participants, 697 groups, 107 topics) for evaluating opinion dynamics in digital-twin role-playing LLM agents, and shows zero-shot agents over-converge while SFT of Llama-3.1-8B partly corrects this.

Problem

Multi-agent role-playing LLM agent simulations of opinion change often show unnatural behavior such as premature convergence, and there are no large empirical benchmarks of real human group interactions to test alignment.

Approach

A benchmark built from multi-round, four-person human deliberation with public tweets, dyadic chats, and private Likert opinions, used to evaluate digital-twin RPLAs via stance-based opinion-dynamics metrics (single-opinion, update, and convergence errors) in next-message and full-dynamics simulation settings.

Key results

• Zero-shot Llama-3.1-8B over-converges in public channel (-0.31 vs human -0.15) and private (-0.41 vs -0.13); SFT moves these to -0.12 and 0.04.
• SFT lowers L2 distance from human transition matrix from 0.84 to 0.51 (public) and 0.83 to 0.30 (private).
• SFT raises public semantic similarity from 0.71 to 0.75 and public ROUGE-L from 0.41 to 0.44 in full dynamics simulation.

Surprising findings

• Private-channel errors exceed public-channel errors across nearly all evaluated systems.
• The fitted DeGroot baseline over-converges even more than the base LLM on public convergence (-0.47).

Limitations

• Data are U.S.-based and not nationally representative.
• Residual over-convergence and large gaps remain, especially in the private channel and full dynamics simulation.
• Only small groups (four participants) and short-horizon discussions are studied.

Experimental setup

Datasets: DEBATE human deliberation data

Benchmarks: DEBATE group-generalization test split DEBATE topic-generalization test split

Metrics: single-opinion prediction error opinion-update prediction error opinion-convergence prediction error semantic similarity ROUGE-L

Baselines: Llama-3.1-8B-Instruct (zero-shot) fitted DeGroot model Uniform LB Noise UB

Builds on: DeGroot model

Closest papers by meaning

• Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs (ICML 2024 · similarity 0.88)
• Multi-LLM Debate: Framework, Principals, and Interventions (NeurIPS 2024 · similarity 0.88)
• Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models? (NeurIPS 2025 · similarity 0.88)
• Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate (NeurIPS 2026 · similarity 0.87)
• AI-Mediated Communication Can Steer Collective Opinion (NeurIPS 2026 · similarity 0.87)
• Emergence of Biased Consensus in Multi-Agent LLM Debates (ICML 2026 · similarity 0.87)