Research map / Multi-agent systems and games / LLM agents
Reflective Multi-Agent Collaboration based on Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COPPER, a multi-agent self-reflection framework where a shared LongChat reflector is fine-tuned via SFT and counterfactual PPO using per-agent counterfactual rewards.
Problem
Improving collaboration in LLM-based multi-agent systems via self-reflection is hard because episode-level rewards do not credit individual agents' reflections (credit assignment), and training separate reflectors per agent scales poorly.
Approach
Compute counterfactual rewards by removing one agent's reflection from the actor prompts and measuring the change in episode return difference, then train a single shared, profile-conditioned reflector with SFT followed by PPO against a regression reward model trained on these counterfactual scores.
Key results
- COPPER improves over initial success rate by 31.8% on HotPotQA, 18.5% on GSM8K, and 86.4% on Checkmate in One Move after 5 trials.
- Applying the GPT-3.5-trained shared reflector to GPT-4 actors yields gains of 27.7%, 9.0%, and 53.3% on the three datasets.
- Fine-tuning Llama-3-8B as the reflector with counterfactual PPO gives a 17.3% enhancement on GSM8K over the initial success rate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the shared reflector with counterfactual PPO | COPPER multi-agent collaboration · HotPotQA | success rate: +31.8% over the initial success rate | Helped |
| Fine-tune the shared reflector with counterfactual PPO | COPPER multi-agent collaboration · GSM8K | success rate: +18.5% over the initial success rate | Helped |
| Fine-tune the shared reflector with counterfactual PPO | COPPER multi-agent collaboration · Checkmate in One Move | success rate: +86.4% over the initial success rate | Helped |
| Remove PPO fine-tuning from the shared reflector (ablation) | COPPER · HotPotQA, GSM8K, and Checkmate in One Move | success rate | Hurt |
| Use episode return difference rewards instead of counterfactual rewards (ablation) | COPPER · HotPotQA, GSM8K, and Checkmate in One Move | success rate | Hurt |
| Use counterfactual rewards to select positive SFT examples instead of episode difference rewards (ablation) | COPPER SFT · HotPotQA, GSM8K, and Checkmate in One Move | success rate | Helped |
| Share one reflector across agents rather than using non-shared reflectors (ablation) | COPPER · HotPotQA, GSM8K, and Checkmate in One Move | success rate | Helped |
| Remove agent profiles from reflector inputs (ablation) | COPPER · HotPotQA, GSM8K, and Checkmate in One Move | success rate | Hurt |
| Apply the shared reflector trained with GPT-3.5 actors to GPT-4 actors | COPPER · HotPotQA | success rate: +27.7% over the initial success rate | Helped |
| Apply the shared reflector trained with GPT-3.5 actors to GPT-4 actors | COPPER · GSM8K | success rate: +9.0% over the initial success rate | Helped |
| Apply the shared reflector trained with GPT-3.5 actors to GPT-4 actors | COPPER · Checkmate in One Move | success rate: +53.3% over the initial success rate | Helped |
| Use Llama-3 as the base shared reflector and fine-tune with counterfactual PPO | COPPER · GSM8K | success rate: +17.3% over the initial success rate | Helped |
Surprising findings
- Reflector fine-tuned with PPO sacrifices early-trial performance for higher final success after five trials, trailing SFT in the first rounds on HotPotQA and GSM8K.
Limitations
- Constructing counterfactual rewards requires additional LLM rollouts, adding data collection cost.
- Long-term memory is restricted to a sliding window with maximum capacity.
- Evaluation uses only 100 randomly sampled test instances per dataset due to compute constraints.
Experimental setup
Datasets: HotPotQA GSM8K Checkmate in One Move ALFWorld
Benchmarks: HotPotQA GSM8K Checkmate in One Move ALFWorld
Metrics: success rate (exact match accuracy) F1 score (HotPotQA reward)
Baselines: CoT ReAct Reflexion (GPT-3.5) Reflexion (LongChat) Retroformer (Multi) COPPER (w/o PPO) COPPER (w/o CF)
Builds on: Reflexion Retroformer ReAct Chain-of-thought PPO
Closest papers by meaning
- Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.88)
- Chain of Agents: Large Language Models Collaborating on Long-Context Tasks (NeurIPS 2024 · similarity 0.87)
- Self-Challenging Language Model Agents (NeurIPS 2025 · similarity 0.87)
- Offline Training of Language Model Agents with Functions as Learnable Weights (ICML 2024 · similarity 0.87)
- Distilling LLM Agent into Small Models with Retrieval and Code Tools (NeurIPS 2025 · similarity 0.87)
- ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaboration (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.