Research map / Multi-agent systems and games / LLM agents
Do as We Do, Not as You Think: the Conformity of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BenchForm, a BBH-derived benchmark with five interaction protocols, to measure conformity of LLMs in multi-agent settings and tests persona and reflection prompts as mitigations.
Problem
Whether LLM agents in multi-agent systems conform to majority or peer opinions, abandoning correct judgments, and what factors influence and mitigate this conformity.
Approach
A conformity-oriented benchmark built from BIG-Bench Hard with Raw, Correct/Wrong Guidance, and multi-round Trust/Doubt protocols inspired by Asch experiments, plus conformity rate (CR) and independence rate (IR) metrics; mitigations are empowered personas and a reflection prompt.
Key results
- Every evaluated LLM shows conformity; e.g., GPT-4o's accuracy drops by 22.6 points under Trust and 13.0 under Doubt protocols.
- Independence rate rises with model size: Qwen2 scales from 19.6% (7B) to 57.6% (72B) IR.
- Enhanced personas raise Llama3-70B IR from 28.6% to 40.0%, and reflection raises it to 68.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Expose LLMs to incorrect group consensus in the Doubt protocol rather than Wrong Guidance | Representative LLMs on BenchForm · BenchForm | conformity rate: 23.5%, 31.3%, and 47.2% | Hurt |
| Increase model size | Qwen2 series · BenchForm | independence rate: 19.6% to 57.6% | Helped |
| Increase previous discussion rounds from 1 to 5 (ablation) | Llama3-70B · BenchForm | conformity rate (Trust): 33.9% to 44.4% | Hurt |
| Increase previous discussion rounds from 1 to 5 (ablation) | Llama3-70B · BenchForm | independence rate: 35.1% to 28.6% | Hurt |
| Increase previous discussion rounds from 1 to 5 (ablation) | Qwen2-72B · BenchForm | independence rate: 61.1% to 57.6% | Hurt |
| Reduce majority size from 6 to 3 (ablation) | Llama3-70B · BenchForm | conformity rate (Doubt): 69.9% to 32.6% | Helped |
| Reduce majority size from 6 to 5 (ablation) | Qwen2-72B · BenchForm | conformity rate (Trust): 30.0% to 39.9% | Hurt |
| Replace the default assistant persona with an empowered persona (ablation) | Llama3-70B · BenchForm | independence rate: 28.6% to 40.0% | Helped |
| Replace the default assistant persona with an empowered persona (ablation) | Qwen2-72B · BenchForm | independence rate: 57.6% to 68.6% | Helped |
| Add a double-checking reflection prompt (ablation) | Llama3-70B · BenchForm | conformity rate (Doubt): 69.9% to 35.2% | Helped |
| Add a double-checking reflection prompt (ablation) | Llama3-70B · BenchForm | independence rate: 28.6% to 68.5% | Helped |
| Add a double-checking reflection prompt (ablation) | Qwen2-72B · BenchForm | conformity rate (Trust): 30.5% to 45.0% | Hurt |
Surprising findings
- Llama3.1 models show strong resistance to wrong guidance but are highly vulnerable under the Doubt protocol (Llama3.1-8B CR^D of 91.2%).
- Reflection increased Qwen2-72B's CR under the Trust protocol (30.5% to 45.0%) because it discarded correct answers to align with the majority.
Limitations
- Only multiple-choice reasoning tasks from BBH are used, limiting generality.
- Closed-source model training details are unknown, so causes of differences cannot be established.
- Real-world settings where agents do not see others' answers before responding are not covered.
Experimental setup
Datasets: BIG-Bench Hard
Benchmarks: BenchForm
Metrics: accuracy conformity rate independence rate
Baselines: GPT-3.5 GPT-4o Llama3.1-405B
Builds on: BIG-Bench Hard Asch conformity experiments
Closest papers by meaning
- Large Language Models Miss the Multi-agent Mark (NeurIPS 2025 · similarity 0.90)
- AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors (ICLR 2024 · similarity 0.89)
- More Capable, Less Cooperative? When LLMs Fail at Zero-Cost Collaboration (ICML 2026 · similarity 0.89)
- Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs (ICML 2026 · similarity 0.89)
- Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation (NeurIPS 2024 · similarity 0.89)
- Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.