Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Peer Learning: Learning Complex Policies in Groups from Scratch via Action Recommendations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces peer learning, where a group of RL agents learns from scratch in separate environments by exchanging action recommendations, with a bandit-style trust mechanism for choosing whose advice to follow.
Problem
Standard RL trains agents in isolation; the paper asks whether agents learning concurrently in a group and sharing action advice can learn complex continuous-control policies from scratch, and how agents should decide whose advice to trust (including against adversarial peers).
Approach
Peers each train in their own copy of the MDP and request action recommendations for their current state from all peers; the choice is made by Boltzmann sampling over trust weights, learned via a critic-based advice score, local per-peer trust values, and/or a global leaderboard, optionally using advantage over own action. Framed as a non-stationary multi-armed bandit.
Key results
- Peer learning with advantage outperforms single-agent RL and Early Advising on 3 of 4 MuJoCo environments (e.g., Walker2d-v4 2160 vs 1558 for single agent).
- On the 21x21 room grid, peer learning (72) outperforms single agent (46), LeCTR (15) and Early Advising (12).
- Averaged over 4 MuJoCo environments, peer learning with all three trust mechanisms (ACT with advantage) reaches 67.1 normalized reward vs 59.7 for single agent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use peer learning with advantage instead of single-agent training | SAC · Walker2d-v4 | average reward over time: 1558(489) -> 2160(346) | Helped |
| use peer learning with advantage instead of single-agent training | SAC · HalfCheetah-v4 | average reward over time: 9270(247) -> 9014(715) | Hurt |
| use peer learning with advantage instead of single-agent training | SAC · Ant-v4 | average reward over time: 2694(660) -> 2459(667) | Hurt |
| use peer learning with advantage instead of single-agent training | SAC · Hopper-v4 | average reward over time: 2059(666) -> 2123(963) | Helped |
| use peer learning instead of single-agent RL | DQN · Room-v21 | average reward over time: 46(16) -> 72(8) | Helped |
| use peer learning instead of single-agent RL | DQN · Room-v27 | average reward over time: 32(13) -> 58(11) | Helped |
| combine at least two of agent values, critic, and trust rather than a single mechanism (ablation) | Peer Learning · 4 MuJoCo environments (averaged) | normalized reward over time | Mixed |
| add advantage to agent values, critic, and trust (ablation) | Peer Learning · 4 MuJoCo environments (averaged) | normalized reward over time: 66.4 -> 67.1 | No clear effect |
| increase the peer group beyond 2 agents | Peer Learning · MuJoCo environments | average reward over time | Helped |
Surprising findings
- Peer learning performance increases with group size and does not saturate at 2 agents, contrary to literature suggesting 2-3 agents are optimal.
- Action advice from peers can improve learning of complex continuous motor skills, in contrast to earlier advice-exchange results.
Limitations
- Trust mechanisms are not a single best choice; no combination significantly dominates across environments.
- Experiments limited to small group sizes (up to about 10) and implementation is not parallel.
- Ant-v4 benefit of communication decreases over time.
Experimental setup
Datasets: MuJoCo HalfCheetah-v4 Walker2d-v4 Ant-v4 Hopper-v4 custom room grid-world
Benchmarks: HalfCheetah-v4 Walker2d-v4 Ant-v4 Hopper-v4 Room-v21 Room-v27
Metrics: average reward over time learning speed
Baselines: Single agent RL LeCTR Early Advising Peer Learning (random advice)
Builds on: Soft Actor-Critic DQN LeCTR Early Advising
Closest papers by meaning
- Fast Peer Adaptation with Context-aware Exploration (ICML 2024 · similarity 0.90)
- Multi-agent cooperation through learning-aware policy gradients (ICLR 2025 · similarity 0.88)
- Cautiously-Optimistic Knowledge Sharing for Cooperative Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.88)
- Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Games (NeurIPS 2024 · similarity 0.86)
- Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning (ICLR 2026 · similarity 0.86)
- Online Learning of Coalition Structures by Selfish Agents (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.