Research map / Multi-agent systems and games / Cooperative multi-agent RL
Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Kaleidoscope, an adaptive partial parameter sharing method for MARL that uses one shared parameter set with learnable STR-based masks per agent, a mask-diversity regularizer, and periodic resetting, extended to critic ensembles.
Problem
Full parameter sharing in cooperative MARL yields homogeneous policies, while non-parameter sharing is sample-inefficient; existing partial sharing schemes are fixed at initialization and not adaptive.
Approach
Learn distinct binary masks per agent over a single shared parameter set using soft threshold reparameterization, encourage pairwise mask discrepancy with a diversity regularizer, and periodically reset parameters masked by all agents; the same masking is applied to critic ensembles to raise the update-to-data ratio.
Key results
- Kaleidoscope outperforms full and non-parameter sharing baselines across eleven scenarios in MPE, MaMuJoCo and SMACv2 (five seeds for MPE/MaMuJoCo, three for SMACv2).
- Averaged test-time FLOPs are reduced to 0.901x (MPE), 0.680x (MaMuJoCo) and 0.890x (SMACv2) relative to FuPS + ID.
- Ablations show diversity regularization contributes most to performance; removing it lets masking degrade results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use weight-level learnable masks instead of neuron masks (ablation) | Kaleidoscope · MPE World and MaMuJoCo Ant-v2-4x2 | performance | Helped |
| Learn masks during training instead of keeping masks fixed (ablation) | Kaleidoscope · MPE World and MaMuJoCo Ant-v2-4x2 | performance | Helped |
| Add diversity regularization between agent masks (ablation) | Kaleidoscope · MPE World and MaMuJoCo Ant-v2-4x2 | performance | Helped |
| Periodically reset parameters masked by all agents (ablation) | Kaleidoscope · MPE World and MaMuJoCo Ant-v2-4x2 | late-stage performance | Helped |
| Use Kaleidoscope parameter sharing for critic ensembles rather than two independent critic parameter sets (ablation) | MATD3 · MaMuJoCo Ant-v2-4x2 | performance | Helped |
| Use layer-wise diversity weights w_l = 2^l instead of w_l = 1 (ablation) | Kaleidoscope critic ensembles · MaMuJoCo | performance | Helped |
| Increase FLOPs via learnable masking relative to FuPS + ID | Kaleidoscope · MPE | normalized FLOPs: 1.0x -> 0.901x | Hurt |
| Increase FLOPs via learnable masking relative to FuPS + ID | Kaleidoscope · MaMuJoCo | normalized FLOPs: 1.0x -> 0.680x | Hurt |
| Increase FLOPs via learnable masking relative to FuPS + ID | Kaleidoscope · SMACv2 | normalized FLOPs: 1.0x -> 0.890x | Hurt |
| Use an excessively small critic diversity coefficient α (ablation) | Kaleidoscope critic ensembles · MaMuJoCo | performance | Hurt |
| Use an excessively large critic diversity coefficient α (ablation) | Kaleidoscope critic ensembles · MaMuJoCo | performance | Hurt |
| Use an overly small actor diversity coefficient β (ablation) | Kaleidoscope · MPE, MaMuJoCo, and SMACv2 | performance | Hurt |
Surprising findings
- Full parameter sharing with agent IDs generally outperforms non-parameter sharing, attributed to higher sample efficiency.
Limitations
- Optimal hyperparameters vary across scenarios; same hyperparameters used for all scenarios.
- Assigning one mask per agent may not scale to hundreds of agents; clustering agents is suggested but not evaluated.
- Environments contain at most 10 agents.
Experimental setup
Datasets: MPE MaMuJoCo SMACv2
Benchmarks: MPE World MPE Push MaMuJoCo Ant-v2-4x2 MaMuJoCo Hopper-v2-3x1 MaMuJoCo Walker2D-v2-2x3 MaMuJoCo Walker2D-v2-6x1 MaMuJoCo HalfCheetah-v2-2x3 MaMuJoCo Swimmer-v2-10x2 SMACv2 Terran_5_vs_5 SMACv2 Protoss_5_vs_5 SMACv2 Zerg_5_vs_5
Metrics: test return FLOPs
Baselines: NoPS FuPS FuPS + ID SePS MultiH SNP
Builds on: MATD3 QMIX soft threshold reparameterization SePS SNP
Closest papers by meaning
- Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.89)
- HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.87)
- Heterogeneous Agent Collaborative Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems (NeurIPS 2026 · similarity 0.86)
- Dynamic Optimizations of LLM Ensembles with Two-Stage Reinforcement Learning Agents (ICML 2026 · similarity 0.86)
- GradPS: Resolving Futile Neurons in Parameter Sharing Network for Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.