Research map / Large language models / Retrieval and knowledge
CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews
Wojciech Kusa, Oscar E. Mendoza, Matthias Samwald, Petr Knoth, Allan Hanbury
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the SLR abstract rather than title or eligibility criteria as the query representation (ablation) | BM25 and MiniLM-L6-v2 · CSMeD-dev-cochrane | NDCG@10: BM25: 0.438 / 0.286 -> 0.503; MiniLM: 0.470 / 0.299 -> 0.517 | Improved |
| Replace BM25 with MiniLM-L6-v2 using abstracts | CSMeD zero-shot retrieval · CSMeD-dev-cochrane | NDCG@10: 0.503 -> 0.517 | Improved |
| Replace BM25 with MiniLM-L6-v2 using abstracts | CSMeD zero-shot retrieval · CSMeD-dev-cochrane | Last Rel: 63.6 -> 65.5 | Hurt |
| Use eligibility criteria rather than abstract as the query representation (ablation) | BM25 and MiniLM-L6-v2 · CSMeD-dev-cochrane | NDCG@10: BM25: 0.503 -> 0.286; MiniLM: 0.517 -> 0.299 | Hurt |
| Use BigBird rather than Longformer architecture (ablation) | Open-domain full-text screening models · CSMeD-ft-test | F1-score: Longformer: 0.398; BigBird-roberta-base: 0.575 | Improved |
| Use BigBird rather than Longformer architecture (ablation) | Clinical full-text screening models · CSMeD-ft-test | F1-score: Clinical-Longformer: 0.435; Clinical-BigBird: 0.609 | Improved |
| Use clinical-domain pretraining rather than open-domain pretraining (ablation) | Longformer and BigBird full-text screening models · CSMeD-ft-test | F1-score: Longformer: 0.398 -> Clinical-Longformer: 0.435; BigBird-roberta-base: 0.575 -> Clinical-BigBird: 0.609 | Mixed |
| Use GPT-4-8k rather than the other evaluated models | Full-text screening · CSMeD-ft-test-small | F1-score: GPT-4-8k-0314: 0.660; the difference is not statistically significant | No clear effect |
| Use GPT-3.5-turbo-16k rather than GPT-3.5-turbo-0301 | Zero-shot full-text screening · CSMeD-ft-test-small | Precision: 0.585 -> 0.712 | Improved |
| Use Clinical-BigBird rather than GPT-3.5-turbo-16k | Full-text screening · CSMeD-ft-test | F1-score: 0.475 -> 0.609 | Improved |
Closest papers by meaning
- Evaluating AI-based Scientific Knowledge Synthesis with Epidemiological Systematic Reviews (NeurIPS 2026 · similarity 0.87)
- SLIDERS: Systematic Reviews via Automated Evidence Synthesis and Reconciliation (NeurIPS 2026 · similarity 0.85)
- CycleResearcher: Improving Automated Research via Automated Review (ICLR 2025 · similarity 0.83)
- From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review (NeurIPS 2025 · similarity 0.82)
- GEAR-Up: Generative AI and External Knowledge-Based Retrieval: Upgrading Scholarly Article Searches for Systematic Reviews (AAAI 2024 · similarity 0.81)
- Are Key-Phrases All That Reviewers Care About? A Comprehensive Benchmarking of Reviewer Matchmaking Systems (AAAI 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
