Research map / Large language models / Instruction and fine-tuning
Democratizing Diplomacy: A Harness for Evaluating Any Large Language Model on Full-Press Diplomacy
Alexander Duffy, Samuel J Paech, Ishana Shastri, Elizabeth Karpinski, Baptiste Alloui-Cros, Tyler Marques, Matthew Lyle Olson
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| progressively more aggressive prompts (V1 to V3) for the game agent (ablation) | Mistral-Small-24B · Diplomacy hold rate | hold rate: 58.9% -> 24.1% | Improved |
| use V3 offensive prompts instead of baseline (ablation) | Devstral-Small · Diplomacy as France | win rate: 3/10 -> 9/10 | Improved |
| use V3 prompts (ablation) | Mistral-Small-24B · Diplomacy support orders | support order success: +18% | Improved |
| remove negotiation (no-press) when comparing o3 with Kimi-K2 (ablation) | o3 · Diplomacy game score | game score gap vs Kimi-K2: +0.65 (p=0.79) -> +3.1 (p=0.021) | Improved |
| use jailbreak command persuasion instead of reasoned argument (ablation) | persuader model vs Mistral-Small · Diplomacy persuasion | allegiance shift success: jailbreak 31%, lies 11% | Improved |
| appeal to reason, empathy or fairness for persuasion (ablation) | persuader model vs Mistral-Small · Diplomacy persuasion | success rate | No clear effect |
| increase model size | LLM players · Diplomacy game score | game score | Improved |
Closest papers by meaning
- Evaluating Language Model Agency Through Negotiations (ICLR 2024 · similarity 0.89)
- Scaling Inference-Time Computation via Opponent Simulation: Enabling Online Strategic Adaptation in Repeated Negotiation (ICML 2026 · similarity 0.88)
- DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy (ICML 2025 · similarity 0.88)
- LLM Active Alignment: A Nash Equilibrium Perspective (NeurIPS 2026 · similarity 0.87)
- Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation (NeurIPS 2024 · similarity 0.87)
- GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
