Research map / Large language models / Agents, code and math
SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SciBench, a college-level scientific problem benchmark (textbook, multimodal, and exam subsets) evaluating GPT-4, GPT-3.5, Claude2, LLaMA-2 and Mistral under CoT, few-shot, and Python/Wolfram tool prompting, plus an LLM-verified error taxonomy of ten skills.
Problem
Existing LLM benchmarks for scientific reasoning are mostly high-school level and require only elementary arithmetic, and are mostly text-only, so they do not reveal the reasoning limits of LLMs on complex college-level science problems.
Approach
A free-response, college-level benchmark drawn from ten textbooks plus a closed exam set, paired with an automated LLM-based self-critique protocol that classifies each incorrect answer into one of ten missing problem-solving skills.
Key results
- Best textbook average is 43.22% for GPT-4 with few-shot Python prompting; GPT-4-Turbo reaches 40.99% zero-shot.
- On the multimodal subset, GPT-4 (PoT) with captions and OCR reaches 13.8% versus 7.4% for the best open model LLaVA-LLaMA-2-13B.
- On the exam subset, GPT-4 averages 57.54%, below the student averages reported for the Data Mining course (80.18% midterm, 72.71% final).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use few-shot prompting with Python instead of zero-shot prompting | GPT-4 on SciBench textbook dataset · SciBench textbook dataset | accuracy (%): 30.36% to 43.22% | Helped |
| Use zero-shot CoT instead of zero-shot prompting (ablation) | GPT-4 on SciBench textbook dataset · SciBench textbook dataset | accuracy (%): 33.79 to 28.52 | Hurt |
| Use few-shot CoT instead of zero-shot prompting (ablation) | GPT-4 on SciBench textbook dataset · SciBench textbook dataset | accuracy (%): 33.79 to 30.36 | Hurt |
| Use few-shot CoT instead of zero-shot prompting (ablation) | GPT-4-Turbo on SciBench textbook dataset · SciBench textbook dataset | accuracy (%): 40.99 to 39.45 | Hurt |
| Use few-shot Python instead of zero-shot prompting (ablation) | GPT-4-Turbo on SciBench textbook dataset · SciBench textbook dataset | accuracy (%): 40.99 to 28.47 | Hurt |
| Use few-shot learning instead of zero-shot learning (ablation) | LLaMA-2-70B on SciBench textbook dataset · SciBench textbook dataset | accuracy (%): 2.41% to 8.40% | Helped |
| Add CoT prompting (ablation) | zero-shot GPT-3.5 on SciBench textbook dataset · SciBench textbook dataset | calculation error rate: 29.0% to 13.6% | Helped |
| Add CoT prompting (ablation) | zero-shot GPT-3.5 on SciBench textbook dataset · SciBench textbook dataset | causal ability error rate: 18.3% to 32.2% | Hurt |
| Add CoT prompting (ablation) | zero-shot GPT-3.5 on SciBench textbook dataset · SciBench textbook dataset | logical decomposition error rate: 18.3% to 25.4% | Hurt |
| Use external tools instead of few-shot CoT (ablation) | GPT-3.5 error profile on SciBench textbook dataset · SciBench textbook dataset | calculation error rate: 14.5% to 6.2% | Helped |
| Use Wolfram Language instead of few-shot CoT (ablation) | Claude2 on SciBench textbook dataset · SciBench textbook dataset | accuracy (%): decrease of 6.70% | Hurt |
| Use Wolfram Language instead of few-shot CoT (ablation) | LLaMA-2-70B on SciBench textbook dataset · SciBench textbook dataset | accuracy (%): decrease of 6.17% | Hurt |
Surprising findings
- GPT-4-Turbo underperforms GPT-4 in few-shot Python prompting, suggesting reduced program-understanding ability.
- Zero-shot CoT increases causal-reasoning errors (32.2% vs 18.3%) even while reducing calculation errors, showing no universally effective prompting strategy.
Limitations
- Error taxonomy relies on an LLM verifier (GPT-3.5) with about 81% agreement with human annotators, and about 20% of classifications were discarded after human review.
- Textbook dataset restricted to single numerical answers rounded to three decimals, excluding symbolic or multi-part answers.
- Only a limited set of models and prompting strategies were evaluated, with temperature zero and no repeated runs.
Experimental setup
Datasets: SciBench textbook dataset (869 problems) SciBench multimodal subset (177 problems) SciBench closed exam dataset (103 problems)
Benchmarks: SciBench textbook dataset SciBench multimodal subset SciBench exam dataset
Metrics: accuracy (5% relative tolerance) exam score error-category distribution
Baselines: GPT-4 GPT-3.5-Turbo Claude2 LLaMA-2-70B Mistral-7B LLaVA-LLaMA-2-13B InternLM-XComposer2-VL Qwen-VL-Plus SPHINX-MoE
Builds on: chain-of-thought prompting program-of-thoughts PAL
Closest papers by meaning
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.90)
- SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models (NeurIPS 2024 · similarity 0.90)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.89)
- PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.89)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.89)
- CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.