Research map / Multi-agent systems and games / Cooperative multi-agent RL

COSAC: Counterfactual Credit Assignment in Sequential Cooperative Teams

NeurIPS 2026 · Multi-agent systems and games › Cooperative multi-agent RL · approach: Coordination graphs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes COSAC, a critic-free per-agent policy gradient for sequential cooperative teams that combines an additive ridge-regression reward decomposition with fictitious counterfactual continuations to estimate a sequential aristocrat-utility advantage.

Problem

Per-agent credit assignment in cooperative teams where agents act in a fixed order and share one team reward; critic-based methods scale poorly, while existing critic-free methods suffer from high variance, importance-sampling blowup, or costly environment replays.

Approach

Extends the aristocrat utility from simultaneous to sequential teams (SeqAU, prefix-conditional baseline), decomposes the advantage into direct and indirect effects under an additive reward model fit by ridge regression, and estimates the indirect effect with policy-only fictitious continuations, requiring no extra reward calls or importance weights.</1-2 sentences>

Key results

• On the sequential bandit at K=16, COSAC's advantage MSE is about 2x, 30x, and 46x smaller than C3, MA-GRPO, and HA-GRPO respectively.
• On ARC with four Qwen3-0.6B agents, COSAC attains the highest learning-curve AUC at every reward shaping alpha (e.g., 0.721 vs 0.663 for MA-GRPO at alpha=0).
• At alpha=0, COSAC's final team reward is 0.910 vs 0.838 for MA-GRPO; at alpha=1 COSAC ties MA-GRPO on final reward.

Surprising findings

• C3 trails the critic-free baselines in regret because its per-iteration cost grows with K, leaving fewer updates under matched budget.
• HA-GRPO degrades rapidly with team size K, consistent with importance-sampling variance blowup.

Limitations

• Assumes an additive surrogate of the team reward; interaction-heavy rewards may need richer decompositions.
• Analysis and experiments restricted to the sequential-bandit regime (one-step episodes); not full sequential MDPs.
• LLM evaluation uses only Qwen3-0.6B on 10 questions, and requires discrete action buckets.

Experimental setup

Datasets: AI2 Reasoning Challenge (ARC)

Benchmarks: AI2 Reasoning Challenge (ARC) sequential cooperative bandit

Metrics: advantage MSE normalized regret AUC of learning curve final team reward

Baselines: MA-GRPO HA-GRPO C3 COMA DOP

Builds on: aristocrat utility COMA DOP HAPPO GRPO

Closest papers by meaning

• Rethinking Credit Assignment in Cooperative MARL via Interventional Reward Response (NeurIPS 2026 · similarity 0.89)
• PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization (NeurIPS 2026 · similarity 0.87)
• Learning to Share: Selective Memory for Efficient Parallel Agentic Systems (ICML 2026 · similarity 0.86)
• Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning (ICLR 2026 · similarity 0.86)
• Shapley-Coop: Credit Assignment for Emergent Cooperation in Self-Interested LLM Agents (NeurIPS 2025 · similarity 0.86)
• Multi-agent cooperation through learning-aware policy gradients (ICLR 2025 · similarity 0.86)