Research map / Large language models / Agents, code and math
Evaluating AI-based Scientific Knowledge Synthesis with Epidemiological Systematic Reviews
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AgentSLR, an open evaluation harness and expert-annotated dataset (16,248 articles from PERG WHO priority-pathogen SLRs) that tests LLMs across SLR stages in epidemiology, finding no single model dominates and extraction is the bottleneck.
Problem
LLM capabilities for systematic literature reviews are underspecified as an evaluation setting, especially for epidemiological evidence synthesis where structured parameter extraction and screening errors can affect public policy.
Approach
A stage-isolated evaluation harness (retrieval, title/abstract and full-text screening, schema-constrained extraction) scored against expert PERG labels, with Flagging/Count/Extraction metrics using partial-credit counts and Jaccard-based bipartite matching of records.
Key results
- Kimi-K2.5 achieves best title/abstract screening F1 of 0.77 and parameter extraction F1 of 0.63; gpt-oss-120b best full-text screening F1 of 0.87 under human-abstract conditioning.
- No model exceeds average field-level extraction F1 of 0.67 across evaluated models.
- gpt-oss-120b reaches average F1 0.70 at USD 13.9 per pathogen run versus USD 1,348.2 for GPT-5.2 (over 96x cost difference).
Surprising findings
- Claude Opus 4.5 and Sonnet 4.5 produced streaming refusals, making them unevaluable.
- Higher cost did not reliably predict higher performance across models.
Limitations
- Evaluation restricted to open-access articles (~27% of PERG corpus), though a 1,004-article closed-access check was broadly comparable.
- English-only screening criteria miss multilingual evidence.
- Meta-analysis and final review writing are not evaluated.
Experimental setup
Datasets: PERG systematic review labels (epireview / priority-pathogen R packages)
Benchmarks: AgentSLR
Metrics: precision recall macro F1 field-level F1 count partial credit flagging F1 cost (USD)
Baselines: GPT-5.2 Claude Opus 4.5 Claude Sonnet 4.5
Builds on: ScreenPrompt PRISMA LLM4SCREENLIT
Closest papers by meaning
- CycleResearcher: Improving Automated Research via Automated Review (ICLR 2025 · similarity 0.87)
- SLIDERS: Systematic Reviews via Automated Evidence Synthesis and Reconciliation (NeurIPS 2026 · similarity 0.85)
- MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks (NeurIPS 2025 · similarity 0.85)
- AAAR-1.0: Assessing AI’s Potential to Assist Research (ICML 2025 · similarity 0.85)
- Can AI Agents Synthesize Scientific Conclusions? (NeurIPS 2026 · similarity 0.85)
- MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.