Research map / Large language models / Retrieval and knowledge
AutoSurvey: Large Language Models Can Automatically Write Surveys
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoSurvey, a multi-phase LLM agent pipeline (retrieval, parallel outline and subsection drafting, refinement, multi-LLM judging) that automatically writes long literature surveys with citations.
Problem
Writing comprehensive literature surveys manually is slow and hard to scale as LLM-related literature grows; naive LLM generation faces context/output length limits, hallucinated references, outdated parametric knowledge, and lacks reliable evaluation metrics.
Approach
A hierarchical pipeline that retrieves relevant papers via embeddings, merges outlines produced over chunks of papers, drafts subsections in parallel with retrieved references, refines for coherence and citation correctness, and scores surveys with a multi-LLM-as-judge protocol calibrated against human experts.
Key results
- At 64k tokens, AutoSurvey reaches 82.25% citation recall and 77.41% precision, versus 68.79% and 61.97% for naive RAG-based LLMs and 86.33%/77.78% for human surveys.
- At 64k tokens AutoSurvey scores 4.73 coverage, 4.33 structure, 4.86 relevance, vs human 5.00, 4.66, 5.00.
- Generation speed for a 64k-token survey is 73.59 surveys per hour versus 0.07 for human writing and 12.56 for naive RAG-based LLMs; the best LLM-judge mixture reaches Spearman rho 0.5429 with human rankings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AutoSurvey instead of naive RAG-based LLM generation | Claude-haiku survey generation · 64k survey | Speed: 12.56 -> 73.59 | Helped |
| Use AutoSurvey instead of naive RAG-based LLM generation | Claude-haiku survey generation · 64k survey | Citation recall: $68.79_{\pm 11.00}$ -> $82.25_{\pm 3.64}$ | Helped |
| Use AutoSurvey instead of naive RAG-based LLM generation | Claude-haiku survey generation · 64k survey | Content quality average: $4.19$ -> $4.62$ | Helped |
| Use AutoSurvey instead of human writing | Survey creation · 64k survey | Speed: $0.07$ -> $73.59$ | Helped |
| Remove retrieval (ablation) | AutoSurvey · Ablation study | Citation recall: $83.48_{\pm 5.05}$ -> $60.11_{\pm 6.42}$ | Hurt |
| Remove retrieval (ablation) | AutoSurvey · Ablation study | Citation precision: $77.15_{\pm 6.05}$ -> $51.65_{\pm 6.33}$ | Hurt |
| Remove retrieval (ablation) | AutoSurvey · Ablation study | Content quality average: $4.57$ -> $4.44$ | Hurt |
| Remove reflection (ablation) | AutoSurvey · Ablation study | Structure: $4.16_{\pm 0.73}$ -> $4.13_{\pm 0.76}$ | Hurt |
| Remove reflection (ablation) | AutoSurvey · Ablation study | Content quality average: $4.57$ -> $4.56$ | No clear effect |
| Use GPT-4 rather than Claude-haiku as the base writer (ablation) | AutoSurvey · Base writer comparison | Content quality average: $4.58$ -> $4.70$ | Helped |
| Use Gemini-1.5-pro rather than Claude-haiku as the base writer (ablation) | AutoSurvey · Base writer comparison | Citation precision: $76.31_{\pm 2.18}$ -> $71.24_{\pm 3.28}$ | Hurt |
| Increase iteration count from 1 to 5 (ablation) | AutoSurvey · Iteration-count comparison | Content quality | Helped |
Surprising findings
- Overgeneralization accounts for 51% of citation errors in a manual analysis, suggesting reliance on parametric knowledge.
- Removing the retrieval mechanism drops citation recall from 83.48 to 60.11, while removing reflection barely changes content quality.
Limitations
- Citation errors remain, with misalignment (39%) and overgeneralization (51%) being the dominant failure types.
- Human-evaluation sample is small (20 surveys, three PhD student evaluators), and LLM-human rank correlation is only moderate.
Experimental setup
Datasets: arXiv computer science papers (530,000-paper retrieval corpus) 20 human-written LLM-topic surveys
Benchmarks: 20 LLM-topic surveys compared to human-written surveys
Metrics: citation recall citation precision coverage structure relevance generation speed Spearman's rho
Baselines: Naive RAG-based LLM generation Human-written surveys
Builds on: retrieval-augmented generation LLM-as-Judge RecurrentGPT STORM
Closest papers by meaning
- AAAR-1.0: Assessing AI’s Potential to Assist Research (ICML 2025 · similarity 0.85)
- CycleResearcher: Improving Automated Research via Automated Review (ICLR 2025 · similarity 0.85)
- Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning (ICLR 2026 · similarity 0.83)
- Automating Thought of Search: A Journey Towards Soundness and Completeness (Student Abstract) (AAAI 2025 · similarity 0.82)
- Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees (NeurIPS 2025 · similarity 0.82)
- AutoData: A Multi-Agent System for Open Web Data Collection (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.