Research map / Multi-agent systems and games / Cooperative multi-agent RL
Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Observation Delays
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Rainbow Delay Compensation (RDC), a MARL framework with a learned compensator (Flash/Echo), delay-reconciled critic, curriculum actor and knowledge distillation, plus a DSID-POMDP model for stochastic individual observation delays, evaluated on MPE and SMAC.
Problem
Observation delays in multi-agent systems are ubiquitous and differ across observation components and agents, violating the Markov assumption and degrading MARL performance under fixed and unfixed delays.
Approach
Formulates DSID-POMDP with per-entity delay distributions and reconstructs delay-free observations via a learned compensator (Flash: one-shot, Echo: autoregressive), combined with a delay-reconciled critic, curriculum learning for the actor, and distillation from a low-delay teacher, all integrated into value-factorization MARL (VDN/QMIX).
Key results
- On MPE TAG with fixed delay 6, Echo+H+DR+KD reaches 182.9 reward versus 84.5 for Oracle and 125.4 for Base, nearly matching delay-free performance.
- Under unfixed delay 3-9 on MPE SPREAD, Echo+H+DR+KD reaches -31.4 reward, outperforming the Oracle at -45.9.
- Compensated models generalize across uniform, binomial, Poisson and normal delay distributions with minor performance fluctuations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add observation compensation, delay-reconciled critic, history input, and knowledge distillation | FT-QMIX · TAG fixed delay 6 | reward: 125.4±22.8 -> 182.5±32.0 | Helped |
| Add observation compensation, delay-reconciled critic, history input, and knowledge distillation | FT-QMIX · SPREAD fixed delay 6 | reward: -40.3±2.4 -> -32.4±2.3 | Helped |
| Add observation compensation, delay-reconciled critic, history input, and knowledge distillation | FT-QMIX · REFERENCE fixed delay 6 | reward: -30.3±1.7 -> -16.7±1.1 | Helped |
| Add history input and knowledge distillation to Flash + DR (ablation) | FT-QMIX with Flash + DR · TAG fixed delay 6 | reward: 150.4±22.5 -> 182.5±32.0 | Helped |
| Add knowledge distillation to Flash + DR + H (ablation) | FT-QMIX with Flash + DR + H · TAG fixed delay 6 | reward: 166.7±29.0 -> 182.5±32.0 | Helped |
| Add history input and knowledge distillation to Flash + DR (ablation) | FT-QMIX with Flash + DR · SPREAD fixed delay 6 | reward: -39.3±1.9 -> -32.4±2.3 | Helped |
| Add knowledge distillation to Flash + DR + H (ablation) | FT-QMIX with Flash + DR + H · SPREAD fixed delay 6 | reward: -33.1±1.9 -> -32.4±2.3 | No clear effect |
| Use Echo instead of Flash in the full method (ablation) | FT-QMIX with RDC · TAG fixed delay 6 | reward: 182.5±32.0 vs 182.9±36.6 | No clear effect |
| Use Echo instead of Flash in the full method (ablation) | FT-QMIX with RDC · REFERENCE fixed delay 6 | reward: -16.7±1.1 vs -16.9±1.2 | No clear effect |
| Enable teacher forcing during compensator training (ablation) | Echo + DR + H · TAG fixed delay 6 | reward: 168.8±23.4 vs 176.9±29.9 | No clear effect |
| Use curriculum learning for the actor (ablation) | FT-QMIX in MPE · MPE tasks | reward | No clear effect |
| Use curriculum learning for the actor (ablation) | FT-QMIX in SMAC · SMAC tasks | win rate | Helped |
Surprising findings
- RDC-enhanced models sometimes exceed the delay-free Oracle, which the authors attribute to extra teacher guidance within the same training horizon.
- Flash compensator inference is much faster (about 0.004 s vs 0.02 s for Echo) but generalizes less well to out-of-distribution delays.
Limitations
- Compensator generalization is weaker on complex SMAC scenarios than on MPE.
- Flash overfits to the history length used in training and generalizes less well to out-of-distribution delays.
- Assumes agents only transmit their own information, so the delay equivalence theorem does not hold and theory relies on strong assumptions.
Experimental setup
Datasets: MPE (simple-tag, simple-spread, simple-reference) SMAC (3s_vs_5z, 5m_vs_6m, 6h_vs_8z)
Benchmarks: MPE simple-tag MPE simple-spread MPE simple-reference SMAC 3s_vs_5z SMAC 5m_vs_6m SMAC 6h_vs_8z
Metrics: episode reward win rate
Baselines: Oracle (delay-free) Base FT-QMIX Base+C Base+DR FT-VDN
Builds on: Dec-POMDP FT-QMIX VDN QMIX delay-reconciled training
Closest papers by meaning
- Addressing Signal Delay in Deep Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Learning Decentralized Partially Observable Mean Field Control for Artificial Collective Behavior (ICLR 2024 · similarity 0.86)
- Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation (NeurIPS 2024 · similarity 0.86)
- Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.86)
- SrSv: Integrating Sequential Rollouts with Sequential Value Estimation for Multi-agent Reinforcement Learning (AAAI 2025 · similarity 0.86)
- ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.