Research map / Large language models / Agents, code and math
How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NegotiationArena, an open-source framework with three multi-turn negotiation games used to benchmark GPT-4, GPT-3.5, Claude-2 and Claude-2.1 against each other and to probe social-behavior prompting and irrational biases.
Problem
Existing LLM benchmarks use static question sets, and it is unclear how well LLM agents negotiate with each other and what irrational or exploitable behaviors they show during negotiation.
Approach
A flexible, serializable game platform with XML-tagged structured messages and hidden reasoning, supporting multi-turn resource exchange, ultimatum and seller-buyer games, plus persona prompts (cunning, desperate) to measure how behavioral tactics change win rates and payoffs.
Key results
- GPT-4 and Claude-2.1 are the strongest negotiators in resource exchange; GPT-4 as second player wins 76% against Claude-2.1 going first.
- Desperate or cunning persona prompts raise GPT-4 second-player win rate substantially; cunning reaches 82% win rate in the ultimatum game but with bimodal payoffs (~49 average).
- Initial price proposal correlates with final accepted price in seller-buyer games (Spearman rho=0.716), indicating anchoring; over-valued buyers counter-propose higher prices 0.41 vs 0.11 for default buyers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the desperate persona prompt to GPT-4 Player 2 | GPT-4 · Multi-Turn Ultimatum | win rate (Player 2): The paper reports that desperate behavior drastically increases Player 2 winning, but gives no exact value. | Helped |
| Add the cunning persona prompt to GPT-4 Player 2 | GPT-4 · Multi-Turn Ultimatum | win rate (Player 2): 82% vs default: almost never | Helped |
| Add the cunning persona prompt to GPT-4 Player 2 | GPT-4 · Multi-Turn Ultimatum | average payoff (Player 2): around 49 vs around 49 | No clear effect |
| Have GPT-4 act as the buyer | GPT-4 · Seller and Buyer | average sale price: $41 across different seller models | Helped |
| Make the buyer over-valued relative to the seller's production cost | GPT-4 buyer · Over-valued Buyer Seller and Buyer | probability of bad counter-proposal P(p1 < p2): 0.11 -> 0.41 | Hurt |
| Prompt the over-valued GPT-4 buyer to be self-interested | GPT-4 buyer · Over-valued Buyer Seller and Buyer | probability of bad counter-proposal P(p1 < p2) | No clear effect |
| Scale up the resources and beliefs while keeping their proportions | GPT-4 buyer and seller · Seller and Buyer | fraction of available money used for sale price | Hurt |
| Change from the classical 2-turn ultimatum to a 3-turn ultimatum | GPT-4 decider · Ultimatum | acceptance probability | Hurt |
| Increase the amount available to split | GPT-4 in the ultimatum game · Ultimatum | fraction of total amount obtained by Player 1: When splitting $10,000,000,000, Player 1 obtains almost 79% | Helped |
Surprising findings
- GPT-4 as a 3-turn ultimatum decider accepts offers in proportion to perceived fairness, unlike its near-deterministic acceptance in the classic 2-turn ultimatum, suggesting memorized rules rather than generalized rationality.
- Over-valued buyers are more likely to counter-propose a higher price than the seller's offer, and prompting to be self-interested does not significantly change this.
Limitations
- Results depend heavily on prompt design, which the authors admit is biased toward Claude's instruction following.
- Weaker models such as GPT-3.5 and Claude-2 frequently break the game format, which can distract stronger agents.
- Llama-2-70B was excluded because it had trouble following instructions.
Experimental setup
Benchmarks: NegotiationArena resource exchange NegotiationArena multi-turn ultimatum NegotiationArena seller and buyer
Metrics: win rate average payoff acceptance probability Spearman correlation
Baselines: GPT-4 GPT-3.5 Claude-2 Claude-2.1
Builds on: Ultimatum game anchoring effect literature
Closest papers by meaning
- PieArena: Ranking and Profiling Language Agents in Realistic Negotiation Scenarios (NeurIPS 2026 · similarity 0.89)
- Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation (NeurIPS 2024 · similarity 0.89)
- Evaluating Language Model Agency Through Negotiations (ICLR 2024 · similarity 0.86)
- EAI: Emotional Decision-Making of LLMs in Strategic Games and Ethical Dilemmas (NeurIPS 2024 · similarity 0.86)
- Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordia (NeurIPS 2025 · similarity 0.85)
- Can Large Language Model Agents Simulate Human Trust Behavior? (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.