Research map / Large language models / Agents, code and math
Can Large Language Models Match the Conclusions of Systematic Reviews?
Christopher Polzak, Alejandro Lozano, Min Woo Sun, James Burgess, Yuhui Zhang, Kevin Wu, Chia-Chun Chiang, Jeffrey J Nirschl, Serena Yeung-Levy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use reasoning model DeepSeek R1 instead of non-reasoning DeepSeek V3 | DeepSeek 671B · MedEvidence | accuracy | Hurt |
| distill reasoning (R1) into Llama 3.3 70B | Llama 3.3 70B · MedEvidence | accuracy | Improved |
| add medical fine-tuning to base LLMs | medical-finetuned models vs base · MedEvidence | accuracy | Hurt |
| increase model size from 7B to 70B | LLM families · MedEvidence | accuracy | Improved |
| increase model size beyond 70B | LLM families · MedEvidence | accuracy | No clear effect |
| use expert-guided prompt instead of basic prompt (ablation) | DeepSeek V3 · MedEvidence | accuracy: nearly 20% drop | Hurt |
| use expert-guided prompt instead of basic prompt (ablation) | five worst models under basic prompt · MedEvidence | accuracy | Improved |
| provide full text instead of abstract only (ablation) | DeepSeek R1 · MedEvidence | accuracy: +4.8% | Improved |
| provide abstracts only instead of full text (ablation) | HuatuoGPT-o1-70B · MedEvidence | accuracy: +24.8% | Improved |
| increase source concordance from 0% to 100% | DeepSeek V3 · MedEvidence | accuracy: 41.21 -> 92.45 | Improved |
| increase number of relevant sources | LLMs · MedEvidence | accuracy | No clear effect |
| increase input token length | LLMs · MedEvidence | accuracy | Hurt |
Closest papers by meaning
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.89)
- Evaluating AI-based Scientific Knowledge Synthesis with Epidemiological Systematic Reviews (NeurIPS 2026 · similarity 0.89)
- Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity (ICML 2025 · similarity 0.88)
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.88)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.88)
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
