Research map / Multi-agent systems and games / Cooperative multi-agent RL
RGMComm: Return Gap Minimization via Discrete Communications in Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RGMComm, a discrete MARL communication method that bounds the return gap between full-observability and discrete-communication policies and trains message generators as online clustering with a cosine-distance regularized information maximization loss.
Problem
Discrete communication in cooperative multi-agent RL under partial observability lacks guarantees on expected return; one-hot messages and Gumbel-softmax heuristics give no quantitative bound on the return gap versus full observability.
Approach
Derives a closed-form upper bound O(sqrt(epsilon) n Q_max) on the return gap, where epsilon is the average cosine distance between joint action-value vectors within each message cluster, and recasts message generation as online clustering of local observations with cluster labels as messages, trained with a cosine-distance locality loss plus a mutual information regularizer.
Key results
- With only |M|=2 message labels (1-bit), RGMComm achieves nearly optimal mean episode reward in Predator-Prey, and with more than 4 labels its reward exceeds the full-observability policy.
- In Cooperative Navigation, RGMComm with 4 message labels yields almost zero return gap, and with 16 labels obtains a higher reward than the full-observability policy.
- RGMComm converges to a higher mean episode reward than CommNet, MADDPG, IC3Net, TarMAC, SARNet and VQ-VIB baselines across 2, 3 and 6 predators.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RGMComm instead of communication baselines | Predator-Prey with 2, 3, and 6 predators · Predator-Prey | mean episode reward | Helped |
| Use RGMComm with 2 message labels (ablation) | RGMComm · Predator-Prey | normalized mean episode reward | Helped |
| Increase the message alphabet size (ablation) | RGMComm · Predator-Prey | normalized mean episode reward | Helped |
| Use more than 4 message labels (ablation) | RGMComm · Predator-Prey | normalized mean episode reward | Helped |
| Use RGMComm with 4 message labels instead of communication baselines | RGMComm · Cooperative Navigation | reward | Helped |
| Use RGMComm with 16 message labels instead of communication baselines | RGMComm · Cooperative Navigation | reward | Helped |
| Use 16 message labels (ablation) | RGMComm · Cooperative Navigation | reward | Helped |
| Use Tanh activation for action-value vectors (ablation) | RGMComm · Activation-function ablation | return | Helped |
| Replace Tanh with Softmax activation (ablation) | RGMComm · Activation-function ablation | return | Hurt |
| Remove the activation function for action-value vectors (ablation) | RGMComm · Activation-function ablation | return | Hurt |
Surprising findings
- With more than 4 message labels, RGMComm's reward exceeds that of the full-observability policy, which the authors attribute to discrete messages giving less noisy signals.
- With 16 labels in Cooperative Navigation, RGMComm achieves higher reward than the full-observability policy.
Limitations
- The theoretical bound assumes discrete finite observation and action spaces; extension to continuous spaces is only sketched.
- Experiments use baselines trained with the same hyperparameters over only 3 seeds, on relatively simple particle environments.
- Interpretability is demonstrated qualitatively in a grid-world maze only.
Experimental setup
Datasets: Multi-Agent Particle Environment
Benchmarks: Predator-Prey (Multi-Agent Particle Environment) Cooperative Navigation Grid-world maze (interpretability)
Metrics: mean episode reward return gap average cosine distance
Baselines: CommNet MADDPG IC3Net TarMAC SARNet VQ-VIB Independent Q-learning Centralized Q-learning
Builds on: Regularized Information Maximization Gumbel-Softmax MADDPG VQ-VIB
Closest papers by meaning
- Multi-Agent Reinforcement Learning with Communication-Constrained Priors (NeurIPS 2025 · similarity 0.90)
- Communication Learning via Backpropagation in Discrete Channels with Unknown Noise (AAAI 2020 · similarity 0.88)
- LLM-Guided Communication for Cooperative Multi-Agent Reinforcement Learning (ICML 2026 · similarity 0.87)
- Exponential Topology-enabled Scalable Communication in Multi-agent Reinforcement Learning (ICLR 2025 · similarity 0.87)
- Efficient Communication in Multi-Agent Reinforcement Learning with Implicit Consensus Generation (AAAI 2025 · similarity 0.86)
- RGMDT: Return-Gap-Minimizing Decision Tree Extraction in Non-Euclidean Metric Space (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.