Research map / Multi-agent systems and games / Cooperative multi-agent RL
HyperMARL: Adaptive Hypernetworks for Multi-Agent RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyperMARL, an agent-conditioned hypernetwork that generates per-agent policy and critic parameters within a fully shared architecture, decoupling observation- and agent-conditioned gradients to enable specialised or homogeneous MARL behaviour without altered objectives.
Problem
Parameter sharing in MARL suppresses behavioural diversity needed for specialisation; coupling agent IDs with observations in shared networks worsens cross-agent gradient interference, and existing remedies add complexity (altered objectives, preset diversity levels, sequential updates).
Approach
An agent-conditioned hypernetwork (linear one-hot or MLP with learned embeddings) generates each agent's actor and critic weights from a single shared parameter vector, so per-agent gradients are averaged before the deterministic agent-specific Jacobian is applied, decoupling observation noise from agent identity.
Key results
- HyperMARL matches NoPS performance and NoPS-level policy diversity on Dispersion, where IPPO/MAPPO FuPS variants struggle to specialise.
- On MAMuJoCo, HyperMARL (MLP) achieves the highest IQM in 3 of 4 scenarios and is the only shared-actor method stable on 17-agent Humanoid.
- HyperMARL exhibits lower actor policy gradient variance than FuPS+ID across 22 scenarios with up to 30 agents; ablation removing gradient decoupling consistently degrades performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HyperMARL instead of FuPS+ID | MAPPO on MAMuJoCo Humanoid-v2 17x1 · MAMuJoCo Humanoid-v2 17x1 | mean episode return (IQM): 566.12 -> 6544.10 | Helped |
| Use HyperMARL instead of FuPS+ID | MAPPO on MAMuJoCo Walker2d-v2 2x3 · MAMuJoCo Walker2d-v2 2x3 | mean episode return (IQM): 4574.39 -> 5064.86 | Helped |
| Use HyperMARL instead of FuPS+ID | MAPPO on MAMuJoCo HalfCheetah-v2 2x3 · MAMuJoCo HalfCheetah-v2 2x3 | mean episode return (IQM): 6771.21 -> 7063.72 | Helped |
| Use HyperMARL instead of FuPS+ID | MAPPO on MAMuJoCo Ant-v2 4x2 · MAMuJoCo Ant-v2 4x2 | mean episode return (IQM): 6148.58 -> 5940.16 | Hurt |
| Remove gradient decoupling by conditioning the hypernetwork on observation and agent ID together (ablation) | HyperMARL · Humanoid-v2 and Dispersion | performance | Hurt |
| Remove reset fan-in/out initialization scaling (ablation) | HyperMARL · Humanoid-v2 | performance | Hurt |
| Remove reset fan-in/out initialization scaling (ablation) | HyperMARL · Dispersion | performance | No clear effect |
| Use a smaller agent embedding (dim 4 rather than dim 64) (ablation) | HyperMARL on 20-agent SMAX · 20-agent SMAX | mean win rate (IQM): 0.1155 -> 0.4455 | Helped |
| Increase embedding dim from 4 to 64 with hidden dims 16 and LR 0.0005 (ablation) | HyperMARL on 20-agent SMAX · 20-agent SMAX | mean win rate (IQM): 0.4455 -> 0.3483 | Hurt |
| Increase hidden dims from 16 to 64 with embedding dim 4 and LR 0.0005 (ablation) | HyperMARL on 20-agent SMAX · 20-agent SMAX | mean win rate (IQM): 0.4455 -> 0.3765 | Hurt |
Surprising findings
- FuPS+ID (No State), which conditions only on agent ID, outperforms standard FuPS+ID in the Specialisation game, showing observation-ID coupling harms specialisation.
- Embedding size matters: in 20-agent SMAX, smaller agent embeddings outperformed larger ones.
Limitations
- Hypernetworks generate large target networks, giving more trainable parameters than NoPS/FuPS for few agents.
- DiCo's diversity levels were hard to tune for larger teams, making baseline comparisons sensitive to hyperparameter choice.
- Linear hypernetworks without a shared bias effectively reduce to independent per-agent policies.
Experimental setup
Datasets: Dispersion (VMAS) Navigation (VMAS) MAMuJoCo SMAX Blind-Particle Spread (BPS) Specialisation and Synchronisation games (self-constructed)
Benchmarks: Dispersion Navigation MAMuJoCo SMAX Blind-Particle Spread
Metrics: IQM of episode return with 95% CI win rate System Neural Diversity (Jensen-Shannon distance) policy gradient variance gradient cosine similarity
Baselines: NoPS FuPS+ID DiCo HAPPO Kaleidoscope SePS
Builds on: IPPO MAPPO QMIX HAPPO SePS
Closest papers by meaning
- Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.91)
- Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learning (AAAI 2025 · similarity 0.89)
- Cooperative Policy Agreement: Learning Diverse Policy for Offline MARL (AAAI 2025 · similarity 0.89)
- JaxMARL: Multi-Agent RL Environments and Algorithms in JAX (NeurIPS 2024 · similarity 0.89)
- ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization (ICLR 2025 · similarity 0.89)
- Dynamic Optimizations of LLM Ensembles with Two-Stage Reinforcement Learning Agents (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.