Research map / Multi-agent systems and games / LLM agents
Learning “Partner-Aware” Collaborators in Multi-Party Collaboration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ICR, a counterfactual-invariance KL-regularized PPO method that trains LLM collaborators to selectively use helpful interventions, evaluated on Wason card and Weights tasks with Llama-3-8B.
Problem
LLM collaborators trained with RLHF or preference optimization tend to uncritically adopt or ignore partner interventions, which hurts common-ground convergence in multi-party collaboration.
Approach
Model collaborator-intervention dynamics as a Modified-Action MDP and regularize the collaborator's policy with a KL term against a counterfactual policy obtained by prepending a prompt that says the intervention will not help, so the policy learns to judge intervention quality without explicit common-ground rewards.
Key results
- On Weights Task full-press, ICR achieves ACC 14.06 vs 9.56 for the next best (DPO), about 47% higher.
- On DeliData full-press, ICR achieves 0.88 accuracy and 3.35 CG, versus 0.82 accuracy for DPO and CG 2.94 for PPO.
- Ablation: lambda_Intent = 0.2 gave the most stable learning; 0.01 and 1.0 both hurt.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with counterfactual invariance regularization (ICR) instead of DPO | Llama 3-8B-Instruct collaborator · Weights Task full-press | ACC: 9.56±0.09 -> 14.06±0.13 | Helped |
| Train with counterfactual invariance regularization (ICR) instead of PPO | Llama 3-8B-Instruct collaborator · Weights Task no-press | ACC: 7.81±0.11 -> 10.87±0.13 | Helped |
| Train with counterfactual invariance regularization (ICR) instead of DPO | Llama 3-8B-Instruct collaborator · DeliData full-press | ACC: 0.82±0.02 -> 0.88±0.02 | Helped |
| Train with counterfactual invariance regularization (ICR) instead of PPO | Llama 3-8B-Instruct collaborator · DeliData full-press | CG: 2.94±0.18 -> 3.35±0.19 | Helped |
| Train with counterfactual invariance regularization (ICR) instead of DPO | Llama 3-8B-Instruct collaborator · DeliData no-press | ACC: 0.79±0.02 -> 0.85±0.02 | Helped |
| Train with counterfactual invariance regularization (ICR) instead of PPO | Llama 3-8B-Instruct collaborator · DeliData no-press | CG: 2.80±0.19 -> 3.18±0.20 | Helped |
| Mask intervention content at evaluation (ablation) | ICR collaborator · Weights Task full-press | ACC: 14.06±0.13 -> 7.23±0.11 | Hurt |
| Use a weaker Llama 3-8B-Instruct intervention agent instead of GPT-4o (ablation) | ICR collaborator · Weights Task full-press | ACC: 14.06±0.13 -> 8.45±0.13 | Hurt |
| Vary counterfactual prompt phrasing during training (ablation) | ICR collaborator · Weights Task full-press | ACC: 14.06±0.13 -> 12.34±0.17 | Hurt |
| Remove counterfactual KL regularization and use counterfactual prompt augmentation with standard PPO (ablation) | ICR training · Weights Task full-press | ACC: 14.06±0.13 -> 8.34±0.16 | Hurt |
| Reduce λIntent from 0.2 to 0.01 (ablation) | ICR collaborator · DeliData no-press | proxy reward | Hurt |
| Increase λIntent from 0.2 to 1.0 (ablation) | ICR collaborator · DeliData no-press | proxy reward | Hurt |
Surprising findings
- Rewarding agents with a consensus signal was counterproductive and reduced task utility or correctness in preliminary experiments.
- Behavior cloning reduced solution diversity (negative CG on DeliData).
Limitations
- Only 8B-scale models trained in a decentralized setting due to compute budget.
- Intervention agent fixed to GPT-4o; generalization to varied or human interventions untested.
- Only two collaborative domains evaluated; agents paired only with similarly trained peers.
Experimental setup
Datasets: DeliData Wason Card Selection Weights Task
Benchmarks: DeliData Wason Card Selection Weights Task
Metrics: ACC common ground gain (CG)
Baselines: BC-Collaborator DPO IPO PPO PSO-Intent
Builds on: PPO DPO IPO PSO-Intent Modified-Action MDP
Closest papers by meaning
- Partner Modelling Emerges in Recurrent Agents (But Only When It Matters) (NeurIPS 2025 · similarity 0.89)
- ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaboration (ICLR 2025 · similarity 0.89)
- CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement (ICML 2026 · similarity 0.88)
- Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning (ICLR 2026 · similarity 0.88)
- Adaptively Coordinating with Novel Partners via Learned Latent Strategies (NeurIPS 2025 · similarity 0.88)
- Beyond Single Stationary Policies: Meta-Task Players as Naturally Superior Collaborators (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.