Research map / Large language models / Retrieval and knowledge
Chain of Agents: Large Language Models Collaborating on Long-Context Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Chain-of-Agents, a training-free framework where sequential worker LLMs pass summaries of short chunks along a chain and a manager LLM produces the final answer, handling long-context QA, summarization, and code completion.
Problem
LLMs struggle with long inputs: input reduction (truncation, RAG) can miss needed information, while window extension suffers from lost-in-the-middle and poor focus on relevant content.
Approach
Split long input into window-sized chunks processed sequentially by worker agents that pass natural-language communication units along a chain, with a separate manager agent synthesizing the final answer, giving each agent a short context while covering the full input.
Key results
- CoA (8k) outperforms Vanilla (8k) and RAG (8k) on all eight QA/summarization/code datasets across text-bison, text-unicorn, and gemini-ultra, with gains up to about 10% or more on several datasets (e.g., NarrativeQA +13.30 points for text-bison-style comparisons, QuALITY +22 for gemini-ultra).
- With Claude 3 models, CoA (8k) beats Vanilla (200k) on NarrativeQA and BookSum, with gains growing from Haiku to Opus (e.g., BookSum improvement 1.66/2.86/3.47 points).
- Multi-path CoA with 5-way permutation and voting/judge improves further, though an oracle selection shows large headroom.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use sequential communicating workers and a manager instead of RAG retrieval | text-bison · NarrativeQA | F1: 14.20 -> 25.26 | Helped |
| use sequential communicating workers and a manager instead of RAG retrieval | text-bison · MuSiQue | F1: 33.83 -> 37.09 | Helped |
| use sequential communicating workers and a manager instead of full-context input | text-bison · GovReport | geometric mean ROUGE: 20.60 -> 26.11 | Helped |
| use CoA with an 8k window instead of Vanilla with a 200k window | claude-3-haiku · NarrativeQA (Average) | score: 7.17 -> 18.80 | Helped |
| remove the manager and have the last worker generate the final response (ablation) | CoA · MuSiQue | F1: 37.09 -> 26.79 | Hurt |
| remove the manager and have the last worker generate the final response (ablation) | CoA · HotpotQA | F1: 53.62 -> 48.58 | Hurt |
| read chunks right-to-left rather than left-to-right (ablation) | CoA · MuSiQue | F1: 37.09 -> 29.77 | Hurt |
| read chunks in random order rather than left-to-right (ablation) | CoA · MuSiQue | F1: 37.09 -> 34.55 | Hurt |
| read chunks in random order rather than left-to-right (ablation) | CoA · HotpotQA | F1: 53.62 -> 56.05 | Helped |
| add a second path reading right-to-left and use an LLM judge (ablation) | CoA · Qasper | F1: 37.17 -> 39.23 | Helped |
| add a second path reading right-to-left and use majority voting (ablation) | CoA · QMSum | geometric mean ROUGE: 16.77 -> 9.51 | Hurt |
| add five left-to-right paths and use majority voting (ablation) | CoA · QuALITY | exact match: 65.42 -> 68.60 | Helped |
Surprising findings
- Vanilla (32k) gains on gemini-ultra are modest, and CoA with an 8k window still beats it, suggesting a small window with communication beats a larger window.
- Parallel multi-agent designs (Merge, Hierarchical) often underperform vanilla single-context baselines, while sequential communication is needed for gains.
Limitations
- Communication between agents is not optimized for LLM-to-LLM messaging; aligned models may not communicate optimally without finetuning or in-context learning.
- Only sequential chain communication explored; debate or richer discussion structures not studied.
- Cost and latency of multiple LLM calls remain high.
Experimental setup
Datasets: HotpotQA MuSiQue NarrativeQA Qasper QuALITY QMSum GovReport BookSum RepoBench-P
Benchmarks: HotpotQA MuSiQue NarrativeQA Qasper QuALITY QMSum GovReport BookSum RepoBench-P
Metrics: F1 exact match ROUGE (geometric mean) code similarity
Baselines: RAG Vanilla (Full-Context) Multi-Agent Voting (Merge) Hierarchical multi-agent Claude 3 long-context (200k)
Builds on: retrieval-augmented generation chain-of-thought position interpolation
Closest papers by meaning
- Fleet of Agents: Coordinated Problem Solving with Large Language Models (ICML 2025 · similarity 0.90)
- LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth (ICML 2026 · similarity 0.90)
- Mixture-of-Agents Enhances Large Language Model Capabilities (ICLR 2025 · similarity 0.89)
- Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models (NeurIPS 2024 · similarity 0.89)
- MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains (ICLR 2026 · similarity 0.89)
- Latent Collaboration in Multi-Agent Systems (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.