Research map / Multi-agent systems and games / Cooperative multi-agent RL

Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems

NeurIPS 2026 · Multi-agent systems and games › Cooperative multi-agent RL · approach: MARL benchmarks and grouping · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Dr. MAS, an agent-wise advantage normalization for GRPO-style RL on cooperative multi-agent LLM systems, plus a shared-GPU-pool framework for co-training heterogeneous LLM agents.

Problem

RL post-training of cooperative multi-agent LLM systems is unstable: a single global GRPO baseline mismatches heterogeneous agents' reward distributions, inflating per-agent gradient norms and causing spikes.

Approach

Normalize each agent's advantages with statistics computed only over the steps where that agent is active (per-agent mean and std), rather than global trajectory-group statistics, which the authors show removes the dominant gradient-inflation term; paired with a multi-LLM training framework that co-locates actor backends on a shared GPU pool and offloads inactive models.

Key results

• On math, Dr. MAS improves over vanilla GRPO by +5.6% avg@16 and +4.6% pass@16 (Qwen3 models).
• On multi-turn search, Dr. MAS improves avg@16 by +15.2% and pass@16 by +13.1%; Qwen2.5-7B non-sharing rises from 28.0/40.5 to 43.8/58.3 under GRPO vs Dr. MAS.
• Ablation on search (Qwen2.5-7B non-sharing): full per-agent (mu_k, sigma_k) reaches 43.8 avg@16 versus 28.0 for global GRPO normalization.

Surprising findings

• Vanilla GRPO on Qwen2.5-7B non-sharing search collapsed to avoiding search-agent calls due to gradient spikes, reaching NaN gradient norm.
• On Qwen3-8B AIME'24 under non-sharing, Stronger-MAS outperforms Dr. MAS (57.0 vs 44.6).

Limitations

• Does not resolve credit assignment across agents and turns.
• Evaluated only on two workflows and two open model families (Qwen, Llama), limiting generality to larger models and other agent architectures.
• Some comparisons (Stronger-MAS) are not apples-to-apples since they include extra search mechanisms.

Experimental setup

Datasets: DAPO-Math NQ HotpotQA

Benchmarks: AIME'24 AIME'25 AMC'23 MATH500 Minerva OlympiadBench NQ TriviaQA PopQA HotpotQA 2WikiMultiHopQA MuSiQue

Metrics: avg@16 pass@16 gradient norm token count API cost estimate

Baselines: GRPO GSPO Stronger-MAS single-agent GRPO

Builds on: GRPO PPO veRL Search-R1 sglang

Closest papers by meaning

• Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs (ICLR 2026 · similarity 0.93)
• Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.91)
• Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
• Dynamic Optimizations of LLM Ensembles with Two-Stage Reinforcement Learning Agents (ICML 2026 · similarity 0.89)
• AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs (NeurIPS 2026 · similarity 0.89)
• ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering (ICML 2026 · similarity 0.89)