Research map / Multi-agent systems and games / Games and equilibria
Reciprocal Reward Influence Encourages Cooperation From Self-Interested Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Reciprocators, first-order RL agents that receive an intrinsic reciprocal reward proportional to an opponent's accumulated value influence, achieving cooperation with naive learners in IPD and Coins without meta-learning or differentiating through opponent updates.
Problem
Cooperation between self-interested RL agents in sequential social dilemmas, where naive learners converge to Pareto-dominated outcomes; existing opponent-shaping methods require privileged access to opponent learning rules, meta-game training, or exponential sample complexity.
Approach
An intrinsic reciprocal reward equal to the product of an influence balance (accumulated value influence of the opponent's actions on the Reciprocator's return, minus the Reciprocator's influence on the opponent) and the Reciprocator's current one-step value influence on the opponent, which encourages tit-for-tat-like shaping of opponent Q-values using only first-order policy gradient training.
Key results
- In IPD-Analytic round robin, Reciprocator achieves -1.06 against itself and against LOLA and M-MAML, resisting exploitation by higher-order methods, though MFOS extorts it slightly (-1.06 vs -0.98).
- In Coins, Reciprocator shapes NL-PPO to collect more of its own coins at rates significantly higher than MFOS, with two Reciprocators reaching roughly 8 reward per 32 steps versus about 2 for LOLA-OM as reported in prior work.
- Reciprocator achieves mutual cooperation in symmetric Coins without self-play, using only first-order PPO with replay-buffer-based influence estimators.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Reciprocator instead of NL | IPD-Analytic against NL · IPD-Analytic vs NL | average reward per episode: -1.98 -> -1.03 | Helped |
| Use Reciprocator instead of LOLA | IPD-Analytic against LOLA · IPD-Analytic vs LOLA | average reward per episode: -1.09 -> -1.05 | Helped |
| Use Reciprocator rather than LOLA-OM | symmetric Coins · Symmetric Coins | average reward per 32 steps: ~ 2 -> ~ 8 | Helped |
| Use Reciprocator rather than MFOS to shape NL-PPO | Coins against NL-PPO · Coins | proportion of own coins collected | Helped |
Surprising findings
- In IPD-Rollout, the Reciprocator's cooperation oscillates as the extrinsic reward reverts it to defection and the reciprocal reward flips sign after exploitation.
- MFOS in Coins does not shape the naive learner toward cooperation but suppresses its learning by collecting coins faster.
Limitations
- Optimizes return within single episodes rather than across episodes, so long-horizon shaping is not captured.
- Reciprocal reward weight is handpicked, creating a suboptimal trade-off with extrinsic return.
- Not evaluated on longer-horizon STORM environments due to compute limitations.
Experimental setup
Datasets: Iterated Prisoner's Dilemma (IPD) Coins
Benchmarks: IPD-Analytic IPD-Rollout Coins
Metrics: average reward per episode probability of cooperation proportion of own coins collected total coin counts
Baselines: Naive Learner (NL) LOLA M-MAML MFOS LOLA-OM Meta-MAPG
Builds on: Value of Interaction LOLA Model-Free Opponent Shaping COMA PPO
Closest papers by meaning
- Multi-agent cooperation through learning-aware policy gradients (ICLR 2025 · similarity 0.88)
- A Unified Model of Direct and Indirect Reciprocity in Multichannel Games (AAAI 2025 · similarity 0.88)
- Self-Play $Q$-Learners Can Provably Collude in the Iterated Prisoner's Dilemma (ICML 2025 · similarity 0.88)
- Fair Cooperation in Mixed-Motive Games via Conflict-Aware Gradient Adjustment (NeurIPS 2025 · similarity 0.87)
- Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Games (NeurIPS 2024 · similarity 0.87)
- Aligning Individual and Collective Objectives in Multi-Agent Cooperation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.