Research map / Large language models / Agents, code and math
CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Research
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CGBench, a benchmark built from ClinGen expert curations that tests LMs on extracting, verifying, and explaining experimental evidence from clinical genetics papers, evaluating 8 LMs with prompting and an LM-as-judge.
Problem
Existing scientific benchmarks focus on narrow, contrived tasks and do not capture real-world expert evidence curation for variant and gene interpretation in clinical genetics.
Approach
Derives three realistic evidence-curation tasks (VCI evidence scoring, VCI evidence verification, GCI experimental evidence extraction) from ClinGen's expert-curated Evidence Repository, with VCEP/SOP-specific guidelines in prompts, and evaluates explanations against curator explanations using an LM judge.
Key results
- On VCI E-Score tertiary codes, o4-mini and Deepseek R1 are best (Precision@5 0.420 and 0.418) while GPT-4o reaches 0.383; reasoning models lead on fine-grained codes.
- On VCI E-Ver, GPT-4o achieves the best F1 (0.634), with Deepseek-R1 at 0.629, and most models over-predict 'met' (positive rate ~0.66 vs. true 0.434).
- On GCI extraction, o4-mini has the best normalized MAE (0.186) and delta strength (0.445), but no model exceeds 50% strength-change accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use reasoning models rather than non-reasoning models for finer-grained tertiary-code prediction | VCI E-Score models · VCI E-Score tertiary | Precision@5: best model (o4-mini) 0.420; best non-reasoning models GPT-4o 0.383 and Llama 4 0.361 | Helped |
| Add description-only in-context examples, increasing the shot count (ablation) | GPT-4o · VCI E-Score tertiary | Precision@5: 0.383 -> 0.435 (5-shot) -> 0.443 (10-shot) -> 0.482 (20-shot) -> 0.469 (30-shot) | Mixed |
| Add description-only in-context examples, increasing the shot count (ablation) | Llama 4 · VCI E-Score tertiary | Precision@5: 0.361 -> 0.382 -> 0.385 -> 0.456 -> 0.469 | Helped |
| Add description-only in-context examples, increasing the shot count (ablation) | o4-mini · VCI E-Score tertiary | Precision@5: 0.420 -> 0.424 -> 0.427 -> 0.463 -> 0.472 | Helped |
| Add description-only in-context examples to primary-code scoring (ablation) | o4-mini · VCI E-Score primary | Precision@5: 0.743 -> 0.816 | Helped |
| Add in-context examples for evidence verification (ablation) | GPT-4o · VCI E-Ver | F1: zero-shot 0.634; best 10-shot 0.657; 20-shot 0.633; 30-shot 0.639 | Mixed |
| Use description-only in-context prompting instead of zero-shot (ablation) | Deepseek R1 · VCI E-Ver | F1 | Hurt |
| Add 30 in-context examples (ablation) | GPT-4o · VCI E-Score explanation agreement | Judge Agreement: 0.486 -> 0.704 | Helped |
| Add 30 in-context examples (ablation) | Llama 4 · VCI E-Score explanation agreement | Judge Agreement: 0.534 -> 0.564 | Helped |
| Add 30 in-context examples (ablation) | o4-mini · VCI E-Score explanation agreement | Judge Agreement: 0.657 -> 0.683 | Helped |
| Summarize text for 10-shot examples and the evaluated input rather than using full text (ablation) | GPT-4o · VCI E-Score tertiary | Precision@5: 0.443 -> 0.425 | No clear effect |
| Summarize text for 10-shot examples and the evaluated input rather than using full text (ablation) | Llama 4 · VCI E-Score | Precision@5: primary 0.857 -> 0.873; secondary 0.485 -> 0.482; tertiary 0.385 -> 0.352 | Hurt |
Surprising findings
- Most models over-predict evidence as 'met' in E-Ver, and o4-mini and Qwen2.5 perform near random on that task despite strong E-Score results.
- In-context examples improved LM-judge explanation agreement, with GPT-4o 30-shot jumping from 0.486 to 0.704, possibly because they match VCI explanation style.
Limitations
- Omits supplementary materials, tables, and figures from the paper context; only PMC open-access full text is used (about 20% of papers).
- LM judge validated only on a small manually reviewed subset and may reward stylistic matching.
- Evidence verification restricted to single-PubMed-ID samples; multi-document reasoning not evaluated.
Experimental setup
Datasets: ClinGen Evidence Repository (VCI and GCI annotations)
Benchmarks: CGBench VCI E-Score CGBench VCI E-Ver CGBench GCI evidence extraction
Metrics: precision@5 recall@5 TPR TNR F1 category precision category recall structure success rate normalized MAE delta strength
Baselines: Random GPT-4o-mini Claude Sonnet 3.7 Qwen2.5 72B o3-mini
Builds on: ACMG/AMP variant classification guidelines chain-of-thought prompting LitGen
Closest papers by meaning
- Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks (ICLR 2026 · similarity 0.86)
- RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation (ICML 2025 · similarity 0.86)
- Contextualizing biological perturbation experiments through language (ICLR 2025 · similarity 0.85)
- DiscoveryBench: Towards Data-Driven Discovery with Large Language Models (ICLR 2025 · similarity 0.85)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.85)
- Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.