Research map / Large language models / Agents, code and math
SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SciEval, a multi-level, multi-disciplinary benchmark (~18k objective and subjective science questions across biology, chemistry, physics) built on Bloom's taxonomy with a dynamically generated subset to limit data leakage, and evaluates 15 LLMs.
Problem
Existing scientific LLM benchmarks rely on pre-collected objective questions that risk data leakage and do not assess subjective research ability or cover science broadly.
Approach
Organize science evaluation into four Bloom's-taxonomy-aligned dimensions (basic knowledge, knowledge application, scientific calculation, research ability), combine static objective questions from community Q&A and public datasets with a dynamic subset generated from PubChem molecules and physics formulas, and add subjective experimental-data questions from university lab courses.
Key results
- GPT-4 achieves the highest static-data average accuracy of 73.93% in the answer-only setting, and is the only model besides GPT-3.5-turbo and Claude-v1.3 above 60%.
- On the physics dynamic subset, most models score near random (about 25% for 4-choice questions); GPT-4 reaches 51.01% accuracy with 3-shot prompting.
- Chain-of-thought helps mainly GPT-series models on static data; on the chemistry dynamic subset, GPT-4 reaches only 11.05% accuracy in the answer-only setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Chain-of-Thought rather than Answer-Only prompting | GPT-4 · Physics (DD) | accuracy: 25.84 -> 17.98 | Hurt |
| Use 3-shot rather than Answer-Only prompting | GPT-4 · Physics (DD) | accuracy: 25.84 -> 51.01 | Helped |
| Use Chain-of-Thought rather than Answer-Only prompting | GPT-3.5-turbo · Physics (DD) | accuracy: 21.80 -> 47.19 | Helped |
| Use Chain-of-Thought rather than Answer-Only prompting | GPT-4 · Chemistry (DD) | accuracy: 11.05 -> 11.65 | Helped |
| Use 3-shot rather than Answer-Only prompting | GPT-4 · Chemistry (DD) | accuracy: 11.05 -> 12.42 | Helped |
| Use 3-shot rather than Answer-Only prompting | Galactica-30B · Chemistry (DD) | accuracy: 0.90 -> 3.30 | Helped |
| Train on a large-scale scientific corpus | Galactica · SciEval, compared with similar-sized LLMs | accuracy | Helped |
| Train on a large-scale scientific corpus | Galactica · Dynamic Data computational problems | performance | Helped |
Surprising findings
- Galactica-30B and Galactica-6.7B, trained on a scientific corpus, outperform similarly sized general LLMs on calculation-type dynamic questions and even some GPT/Claude models on those.
- GPT-4 performs worse with chain-of-thought than answer-only on the physics dynamic subset (17.98%), reportedly due to incorrect formulas.
Limitations
- Experimental data is evaluated manually, which limits scalability and reproducibility.
- Many open models are excluded from the experimental subset due to context length limits.
- Static data relies on GPT-4-generated distractors and answers, with manual checking only.
Experimental setup
Datasets: Socratic Q&A MedQA (USMLE) PubMedQA ChemLLMBench Reagent Selection PubChem University basic science experiment courses
Benchmarks: SciEval
Metrics: accuracy BLEU MSE exact match manual score
Baselines: MMLU AGIEval C-Eval Galactica Vicuna Alpaca
Builds on: Bloom's taxonomy chain-of-thought
Closest papers by meaning
- TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planning (AAAI 2025 · similarity 0.92)
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
- KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR 2024 · similarity 0.90)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.90)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.90)
- RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.