Research map / Multi-agent systems and games / LLM agents

Opponent Shaping in LLM Agents

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ShapeLLM, a model-free opponent shaping method for transformer LLM agents, and shows a gemma-2-2b-it shaper can exploit or guide co-players in repeated 2x2 games (IPD, IMP, ICG, ISH, C-IPD).

Problem

Whether LLM agents, like RL agents, can shape co-players' learning dynamics through interaction alone, given that existing opponent shaping methods (LOLA, M-FOS, SHAPER) need higher-order derivatives, dual agents, or RNN architectures unavailable in transformers.

Approach

Adapts model-free opponent shaping to transformers by putting history (most recent joint action) and context (compressed cumulative state visitation counts across episodes) into natural-language prompts, and trains the shaper with PPO at the trial level while naive LLM opponents learn with PPO between episodes.

Key results

• In the IPD, the shaper achieves 3.96 average reward per step vs 0.10 for the opponent, versus 1.00 each in the naive-learner baseline.
• In the cooperative IPD, shaping yields mutual cooperation with rewards 5.88 (shaper) and 2.86 (opponent), versus 1.00 each in baseline.
• In the Iterated Stag Hunt, shaping moves runs to Pareto-optimal stag-stag (about 3.96 each), while 90% of baseline runs converge to hare-hare (1.30 each).

Surprising findings

• An enriched-observation naive learner appeared to shape in the ICG under one initialization, but a control showed this was driven by initialization rather than observation space alone.

Limitations

• Only one small model (gemma-2-2b-it) evaluated, so scaling to larger models is untested.
• Agents restricted to fixed action tokens rather than natural language communication.
• Only 2x2 matrix games studied.

Experimental setup

Benchmarks: Iterated Prisoner's Dilemma Iterated Matching Pennies Iterated Chicken Game Iterated Stag Hunt Cooperative IPD

Metrics: average reward per step state visitation frequency

Baselines: naive learners (independent PPO) enriched-observation naive learner

Builds on: LOLA M-FOS SHAPER PPO QLoRA

Closest papers by meaning

• Advantage Alignment Algorithms (ICLR 2025 · similarity 0.89)
• Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation (NeurIPS 2024 · similarity 0.88)
• MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games (ICML 2026 · similarity 0.88)
• $\texttt{Multi}^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments (ICML 2026 · similarity 0.88)
• MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs (ICLR 2026 · similarity 0.88)
• Evaluating LLMs in Open-Source Games (NeurIPS 2025 · similarity 0.87)