Research map / Multi-agent systems and games / LLM agents
Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CORY, which duplicates an LLM into a pioneer and an observer trained with PPO under a shared collective reward, with knowledge transfer and periodic role exchange, and reports better policy optimality and less distribution collapse than PPO on IMDB sentiment completion and GSM8K.
Problem
RL fine-tuning of LLMs with PPO is unstable under sparse rewards and large action spaces, often leading to distribution collapse (over-optimization far from the pre-trained policy).
Approach
Extend single-agent RL fine-tuning to a sequential cooperative two-agent setting: a pioneer answers the query, an observer answers conditioned on the query and the pioneer's answer, both share the sum of task rewards, and their roles are swapped periodically so each model learns both prompt formats.
Key results
- On IMDB, CORY reaches similar task reward to single-PPO after 100 iterations while the KL divergence of single-PPO is more than twice that of CORY.
- On GSM8K, the CORY fine-tuned 4-bit Llama-2-7b-chat reaches pass@1 of 18%, and CORY's task reward curve is more stable than single-PPO, whose reward peaks around 50 iterations and then declines.
- Ablation on IMDB shows that doubling model size with single-PPO (GPT2-XL) leads to rising KL divergence and distribution collapse, so CORY's gains are not simply due to more parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CORY instead of single-agent PPO | Llama-2-7B-chat fine-tuning · GSM8K | task reward | Helped |
| Use CORY instead of single-agent PPO | Llama-2-7B-chat fine-tuning · GSM8K | KL divergence | Helped |
| Use CORY instead of single-agent PPO | GPT2-Large fine-tuning · IMDB Review | task reward | No clear effect |
| Use CORY instead of single-agent PPO | GPT2-Large fine-tuning · IMDB Review | KL divergence | Helped |
| Increase single-PPO model size from GPT2-Large to GPT2-XL (ablation) | single-agent PPO · IMDB Review | KL divergence | Hurt |
| Disable knowledge transfer while retaining role exchange and collective task reward (ablation) | CORY · IMDB Review | task reward stability | Hurt |
| Disable knowledge transfer while retaining role exchange and collective task reward (ablation) | CORY · IMDB Review | KL divergence | Hurt |
| Disable role exchange while retaining knowledge transfer (ablation) | CORY · IMDB Review | KL divergence | Hurt |
| Use CORY instead of PPO with learning rate 1e-4 | Llama-2-7B-chat fine-tuning · GSM8K | training stability | Helped |
| Use CORY instead of PPO with learning rate 1e-5 | Llama-2-7B-chat fine-tuning · GSM8K | training stability | Helped |
Surprising findings
- Scaling single-agent PPO to GPT2-XL (1.5B) did not reduce KL divergence and caused distribution collapse, suggesting the gain is not from parameter count.
- The observer in CORY ends up with significantly lower KL divergence than the pioneer when role exchange is disabled.
Limitations
- Evaluation limited to two tasks and small models (GPT-2 variants, Llama-2-7b/13b chat), with GSM8K pass@1 of only 18%.
- Relies on the collective reward being the sum of two agents' task rewards and only compares against PPO, not other RL algorithms.
- Mostly qualitative or single-run curves; statistical significance and seeds not reported.
Experimental setup
Datasets: IMDB Review GSM8K
Benchmarks: IMDB Review GSM8K
Metrics: task reward (sentiment score) KL divergence combined reward pass@1 pass@k
Baselines: PPO PPO-GPT-2-xl PPO-13b
Builds on: PPO TRL StackLlama
Closest papers by meaning
- Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems (NeurIPS 2026 · similarity 0.90)
- FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents (ICML 2026 · similarity 0.89)
- $\texttt{Multi}^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments (ICML 2026 · similarity 0.89)
- Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.89)
- Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs (ICLR 2026 · similarity 0.88)
- Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.