Research map / Multi-agent systems and games / Cooperative multi-agent RL

Heterogeneous Agent Collaborative Reinforcement Learning

NeurIPS 2026 · Multi-agent systems and games › Cooperative multi-agent RL · approach: MARL benchmarks and grouping · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HACRL, a setting where heterogeneous LLM agents share verified RLVR rollouts during training and run independently at inference, and HACPO, a GSPO-based algorithm with capability-aware baselines, exponential importance sampling and stepwise clipping.

Problem

Multiple heterogeneous LLM agents solving the same RLVR task train independently, wasting costly on-policy rollouts that could be shared across agents for mutual improvement.

Approach

Shared cross-agent rollouts are reweighted with a capability ratio in the advantage baseline, a capability-discrepancy coefficient on gradients, a non-gradient exponential reweighting of cross-agent sequence-level importance ratios, and a stepwise tightening of cross-agent clipping bounds.

Key results

• HACPO improves average accuracy over GSPO x2 by +0.064 (Qwen3-4B), +0.019 (Qwen3-4B-Instruct), +0.028 (Qwen3-1.7B-Base), +0.026 (Qwen3-4B-Base), +0.022 (Qwen3-4B with Qwen3-8B-Base style pairs) and +0.056 (Llama3.2-3B pairing), with the paper claiming 3.6% average gain at half rollout cost.
• Ablation: removing agent-capability-aware advantage drops 1.7B-Base average from 0.493 to 0.465 and 4B-Base from 0.601 to 0.586.
• Naive rollout sharing without the proposed mechanisms underperforms GSPO on all heterogeneity settings (e.g., 4B-Instruct 0.691 vs GSPO 0.799).

Surprising findings

• Naive rollout sharing without the proposed corrections substantially underperforms single-agent GSPO, showing that heterogeneous rollouts without mitigation can hurt training.
• Cross-agent importance ratios shift strongly (mean about 0.896 vs about 1.0 for self-generated), so naive sequence ratios are unreliable.

Limitations

• Empirical evaluation is restricted to two-agent settings; the n>=3 setting is not tested.
• Gains are smaller for some combinations, e.g., AIME2025 and Minerva show mixed results, and the 1.7B/Llama results are modest on some benchmarks.
• Hyperparameters (alpha, delta, delta_step) vary per model combination and must be tuned.

Experimental setup

Datasets: MATH

Benchmarks: MATH-500 MATH GSM8K AIME2025 AMC23 Minerva Olympiad

Metrics: accuracy average accuracy

Baselines: GRPO GSPO GSPO x2 Naive rollout sharing

Builds on: GSPO GRPO DAPO verl

Closest papers by meaning

• Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems (NeurIPS 2026 · similarity 0.88)
• Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments (ICML 2024 · similarity 0.88)
• Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs (ICLR 2026 · similarity 0.87)
• HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.87)
• Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
• Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.86)