Research map / Multi-agent systems and games / Games and equilibria

Differentiable Belief-based Opponent Shaping

NeurIPS 2026 · Multi-agent systems and games › Games and equilibria · approach: Opponent modeling in competitive games · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes D-BOS, a first-order opponent-shaping method that backpropagates through k-step softmax-Bayes belief updates of observers, added as a correction to PPO, and evaluated on hidden-role games (RTG, Avalon, CoinGame).

Problem

Opponent shaping methods act in high-dimensional parameter/policy/value spaces, while belief-manipulation methods (BBM) use myopic hand-coded deception objectives; the paper asks how to shape observers' beliefs over multiple steps driven by task reward in hidden-role games.

Approach

Treat each observer's posterior over the shaper's hidden role as the opponent state, unroll k differentiable softmax-Bayes belief updates, and inject the resulting belief-shaping gradient (via a learned belief critic) into PPO, so the direction of shaping is set by environment reward rather than a prescribed deception objective.

Key results

• Avalon: D-BOS k=1 with estimated observer model reaches win rate 0.557 ± 0.048 vs PPO 0.343 ± 0.032 and BBM 0.243 ± 0.098.
• Rescue-the-General: D-BOS k=3 achieves win rate 0.205 ± 0.138 vs PPO 0.101 ± 0.042 and BBM 0.047 ± 0.011; k=5 collapses to 0.066.
• CoinGame: D-BOS gains are modest; k=5 gives -4.227 ± 0.848 coin return vs PPO -4.960 ± 0.147, with k=1 and k=3 near PPO.

Surprising findings

• BBM underperforms vanilla PPO in both Avalon and RTG, suggesting hard-coded myopic deception objectives are brittle.
• Longer belief horizon (k=5) collapses performance in RTG, while k=3 is best there, indicating compounding ToM error.

Limitations

• Evaluation is against co-trained opponents; generalization to held-out or independently trained opponents is untested.
• Shaping signal can be small or noisy relative to policy gradient; sensitive to lambda, critic quality and gradient scale.
• Error bound grows with horizon k and ToM approximation error; estimated-observer mode adds learned modules whose error is uncontrolled.

Experimental setup

Datasets: Rescue-the-General Avalon5 Multi-agent CoinGame

Benchmarks: Rescue-the-General Avalon5 Multi-agent CoinGame

Metrics: win rate coin return

Baselines: PPO BBM

Builds on: LOLA Bayesian Belief Manipulation (BBM) PPO Advantage Alignment QMDP-net

Closest papers by meaning

• Opponent Shaping in LLM Agents (ICLR 2026 · similarity 0.87)
• Opponent Modeling based on Subgoal Inference (NeurIPS 2024 · similarity 0.86)
• Advantage Alignment Algorithms (ICLR 2025 · similarity 0.86)
• Action-Dependent Optimality-Preserving Reward Shaping (ICML 2025 · similarity 0.85)
• Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Games (NeurIPS 2024 · similarity 0.85)
• State-Constrained Zero-Sum Differential Games with One-Sided Information (ICML 2024 · similarity 0.85)