Research map / Large language models / Instruction and fine-tuning

DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy

ICML 2025 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Fine-tunes Llama 3 8B with LoRA on unit-level autoregressive action data labeled by piKL-Hedge Q-values so it learns an equilibrium-style policy for no-press Diplomacy, beating Cicero with 1.5% of its data.

Problem

Equilibrium search for Diplomacy needs huge compute to generate training data, and LLM agents struggle with the exponential joint action space (up to ~34 units, ~26 actions each) and strategic interactions among seven players.

Approach

Factorize the joint multi-unit action into sequential unit-level decisions with an LLM, define unit-level Q-values from piKL-Hedge equilibrium (approximated by a log-sum-exp lower bound over sampled joint actions), and fine-tune the LLM with a Q-weighted log-likelihood loss.

Key results

• In a baseline population, DipLLM achieves 23.0% SoS score and 22.3% win rate vs Cicero's 20.8% and 20.5%.
• Ablation: AF plus fine-tuning reaches 29.4% SoS vs 0.2% for neither technique, 9.9% for AF only, and 0.8% for fine-tuning only, against six DipNet agents.
• DipLLM with 500 fine-tuning games outperforms DipNet fine-tuned on the same data by 6.7%.

Measured effects reported in the paper

Surprising findings

• Fine-tuning a one-step (non-factorized) LLM on joint actions yields almost no gain (0.8% SoS), so factorization is essential.
• Against OpenAI-o3-mini, DeepSeek-R1 and GPT-4o in an appendix population, DipLLM reaches 60.7% SoS, far above these LLMs.

Limitations

• Fine-tuning data is generated by external models (DipNet plus piKL-Hedge) rather than self-play.
• Search-enhanced variant only preliminarily explored due to compute constraints.
• Evaluation is no-press Diplomacy only, with opponents from a limited population.

Experimental setup

Datasets: DipNet-generated Diplomacy games labeled with piKL-Hedge Q-values

Benchmarks: no-press Diplomacy 1v6 games (SoS score, win rate, most SCs, survived, defeated)

Metrics: sum-of-squares score win rate most supply centers survived rate defeated rate

Baselines: Cicero DORA DNVI DipNet Prompt (Richelieu-style) SFT OpenAI-o3-mini DeepSeek-R1 GPT-4o

Builds on: piKL-Hedge DipNet Cicero LoRA Richelieu

Closest papers by meaning

• Opponent Shaping in LLM Agents (ICLR 2026 · similarity 0.87)
• Scaling Inference-Time Computation via Opponent Simulation: Enabling Online Strategic Adaptation in Repeated Negotiation (ICML 2026 · similarity 0.87)
• From Debate to Equilibrium: Belief‑Driven Multi‑Agent LLM Reasoning via Bayesian Nash Equilibrium (ICML 2025 · similarity 0.86)
• MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games (ICML 2026 · similarity 0.86)
• LLM-PySC2: Starcraft II learning environment for Large Language Models (NeurIPS 2025 · similarity 0.86)
• Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation (NeurIPS 2024 · similarity 0.86)