Research map / Multi-agent systems and games / LLM agents

MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains Qwen3-4B via multi-agent self-play RL in cooperative and competitive games, using turn-level sum-then-normalize advantages and agent-specific normalization, and shows gains on math/QA benchmarks inside MAD and AutoGen.

Problem

Extending RL for LLMs to multi-turn, multi-agent settings is hard due to long-horizon credit assignment and heterogeneous, role-dependent advantage variance; the paper asks whether strategic-game self-play can improve general multi-agent reasoning.

Approach

Sum-then-normalize turn-level advantage (Monte Carlo cumulative returns centered by batch mean, equivalent to GAE with gamma=lambda=1 and an empirical-mean baseline), computed separately within each player-role subgroup, trained via self-play on six two-player games with GRPO-style updates.

Key results

• Generalist MARSHAL agent reaches 28.7% improvement on Leduc Hold'em and 22.9% on Simple Hanabi over Qwen3-4B in held-out games.
• In the AutoGen cooperative framework, the MARSHAL generalist improves AIME by 10.00% zero-shot, with average gain of about 3.5% across benchmarks in MAS settings.
• In MAD, the generalist achieves 7.57% gain on GPQA-Diamond and average 75.96 vs 72.45 for Qwen3-4B.

Surprising findings

• Fixed-opponent training causes severe overfitting; the Kuhn Poker fixed-opponent specialist scores 0 on several games outside its domain.
• Agent-specific advantage normalization matters mainly for competitive games with distinct role return distributions and has only mild effect on the cooperative Hanabi specialist.

Limitations

• Only two-player games are studied; scaling to N-player settings is left open.
• Gains in MAS are modest on average (about 3.5%) and evaluated on a small number of problems (e.g., AIME 30 problems), with a single seed evident from reported results.
• Relies on fixed game-specific opponents for evaluation and a limited set of six games.

Experimental setup

Datasets: Tic-Tac-Toe Kuhn Poker Mini Hanabi Connect Four Leduc Hold'em Simple Hanabi MATH500 GSM8K AQUA-RAT AIME24 AMC23 MMLU-STEM

Benchmarks: MATH500 GSM8K AQUA-RAT AIME24 AMC23 MMLU-STEM GPQA-Diamond Connect Four Leduc Hold'em Simple Hanabi

Metrics: normalized game return accuracy average score

Baselines: Qwen3-4B SPIRAL MT-GRPO fixed-opponent training

Builds on: GRPO PPO Generalized Advantage Estimation SPIRAL MT-GRPO

Closest papers by meaning

• Sparks of Cooperative Reasoning: LLMs as Strategic Hanabi Agents (ICML 2026 · similarity 0.90)
• Strat-Reasoner: Reinforcing Strategic Reasoning of LLMs in Multi-Agent Games (ICML 2026 · similarity 0.89)
• BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.89)
• MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games (ICML 2026 · similarity 0.89)
• Multi-agent KTO: Enhancing Strategic Interactions of Large Language Model in Language Game (NeurIPS 2025 · similarity 0.88)
• GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations (NeurIPS 2024 · similarity 0.88)